Content recommendation based on collections of entities

ABSTRACT

Techniques for content recommendation are described. Some embodiments provide a content recommendation system (“CRS”) configured to recommend content items that are related to a collection of entities. A content item may be considered related to a collection of entities based on various factors, including whether and how often the article references or otherwise covers the entities of the collection, the size of the article, other entities that are covered by the article but that are not in the collection, article recency, or article credibility. Recommending content items may also or instead include determining entities that are related to a collection. An entity can be considered related to a collection based on various factors, such as whether the entity is of the same or similar type to entities of the collection, or whether the entity appears in some article in a relationship with one or more entities of the collection.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/038,192 filed on Mar. 1, 2011, which claims priority from U.S.Provisional Patent Application No. 61/309,318 filed on Mar. 1, 2010, thecontents of which applications are hereby incorporated herein byreference in their entireties.

TECHNICAL FIELD

The present disclosure relates to methods, techniques, and systems forrecommending content and, in particular, to methods, techniques, andsystems for recommending content items that are related to a collectionof entities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example screen display provided by an exampleembodiment of a content recommendation system.

FIG. 2 illustrates an example block diagram of an example embodiment ofa content recommendation system.

FIGS. 3A-3I illustrate example screen displays provided by an exampleembodiment of a content recommendation system.

FIGS. 4A-4D illustrate additional example screen displays provided by anexample embodiment of a content recommendation system.

FIG. 5 is an example block diagram of an example computing system forimplementing a content recommendation system according to an exampleembodiment.

FIG. 6 is an example flow diagram of a content recommendation processperformed by an example content recommendation system according to anexample embodiment.

FIG. 7 is an example flow diagram of an article recommendation processperformed by an example content recommendation system according to anexample embodiment.

FIG. 8 is an example flow diagram of an entity recommendation processperformed by an example content recommendation system according to anexample embodiment.

FIG. 9 is an example flow diagram of a query generation processperformed by an example content recommendation system according to anexample embodiment.

FIG. 10 is an example flow diagram of an article scoring processperformed by an example content recommendation system according to anexample embodiment.

FIGS. 11A-11E illustrate example screen displays generated by queriesexecuted for content recommendation provided by an example embodiment ofa content recommendation system.

DETAILED DESCRIPTION

Embodiments described herein provide enhanced computer- andnetwork-based methods and systems for recommending content. Exampleembodiments provide a content recommendation system (“CRS”) configuredto recommend content items that are related to a collection of entities.A CRS may be used, for example, by a user to locate or follow pertinentcontent from a large corpus of documents, such as from news articlesavailable via a wide-area or local area networked environment, to a setof entities of personal interest. Content items may include articles andinformation about or contained in articles or other documents. A typicalarticle comprises text, audio, image, and/or video data that providesinformation about one or more entities and relationships between thoseentities. Information about or contained in articles thus includesentities and relationships referenced or otherwise covered by anarticle. Entities include people, places (e.g., locations),organizations (e.g., political parties, corporations, groups), events,concepts, products, substances, and the like. Table 3, below, includes alist of example entity types. Fewer or more can be made available.Information about or contained in articles may also or instead includevarious types of meta-information, such as article titles, authors,publication dates, summaries, quotations, and the like.

In one embodiment, a user can interact with the CRS to define acollection of entities of interest. The CRS then recommends contentitems that are related to the entities in the user's collection.Recommending content items may include determining one or more articles,such as news stories, Web pages, Blog posts, or the like, that arerelated to the collection. An article can be considered related to acollection based on various factors, including whether and how often thearticle references or otherwise covers the entities of the collection,the size of the article, other entities that are covered by the articlebut that are not in the collection, article recency, articlecredibility, and the like. Recommending content items may also orinstead include determining one or more entities that are related to thecollection. An entity can be considered related to a collection based onvarious factors, such as whether the entity is of the same or similartype to entities of the collection, whether the entity appears in somearticle in a relationship with one or more entities of the collection,and the like. Specific example techniques for determining relatedarticles and entities are discussed, for example, with respect to FIGS.7-11, below.

FIG. 1 illustrates an example screen display provided by an exampleembodiment of a content recommendation system. In particular, FIG. 1illustrates a Web browser 100 that displays a screen 102 (e.g., a Webpage) that is being used by a user to interact with the contentrecommendation system. The screen 102 displays information about andrelated to a collection named “Collection1,” including a collectioncontents section 104, a related entities section 106, and a relatedarticles section 108. The collection contents section 104 indicates thatCollection1 includes two entities: Roger Moore and Sean Connery. Therelated articles section 108 includes references to multiple articlesthat are related to either or both of the entities of Collection1. Forexample, article reference 110 includes a title, a snippet from thearticle, and indications of at least some of the entities (e.g., RichardDemarco, Edinburgh, The Scotsman, Joseph Beuys, Sean Connery, Scotland,etc.) referenced in the article. The related entities section 106includes references to multiple entities that are related to theentities in Collection1, including James Bond, George Lazenby, PierceBrosnan, Timothy Dalton, Daniel Craig, and the like.

Some of the examples discussed herein focus primarily on collectionsthat include indications of one or more singular entities (e.g., theactor Sean Connery). In other embodiments, a collection can includeindications that reference multiple entities, such as by including namesthat ambiguously reference multiple entities (e.g., Jaguar the animal orJaguar the car); facets (e.g., more finely granular characteristics ofentities such as categories, types, and/or characteristics, such asactor, politician, nation, drug, automobile, and the like); keyterms(e.g., one or more words or phrases, such as “bond actors,” “007,” andthe like); and the like. Table 4, below, includes a list of examplefacets for the various entity types. Fewer or more can be madeavailable. The techniques described herein are equally applicable tocollections that include such indications of multiple and/or ambiguousentities.

FIG. 2 illustrates an example block diagram of an example embodiment ofa content recommendation system. In particular, FIG. 2 illustrates acontent recommendation system (“CRS”) 200 that includes a contentingester 211, an entity and relationship identifier 212, an entityrecommender 213, an article recommender 214, and a data store 217. Thedata store 217 includes a document store 217 a, an entity store 217 b, acollection store 217 c, and a relationship index 217 d. The contentingester 211 receives and indexes content from various content sources255, including sources such as Web sites, Blogs, news feeds, videofeeds, and the like. The content ingester 211 may also receive contentfrom non-public or semi-public sources, including subscription-basedinformation services, access-controlled social networks, and the like.The content ingester 211 provides content information, including dataincluded within content items (e.g., text, images, video) and meta-dataabout content items (e.g., author, title, date, source), to the entityand relationship identifier 212. The content information may be provideddirectly (as illustrated) and/or via some intermediary, such as the datastore 217.

The entity and relationship identifier 212 receives content informationfrom the content ingester 211 and identifies entities and relationshipsthat are referenced therein. Various automatic and semi-automatictechniques are contemplated for identifying entities within contentitems. In one embodiment, the identifier 212 uses natural languageprocessing techniques, such as parts of speech tagging and relationshipsearching, to identify sentence components such as subjects, verbs, andobjects, and to disambiguate and identify entities. Example relationshipsearching technology, which uses natural language processing todetermine relationships between subjects and objects in ingestedcontent, is described in detail in U.S. patent application Ser. No.11/012,089, filed Dec. 13, 2004, and entitled “METHOD AND SYSTEM FOREXTENDING KEYWORD SEARCHING FOR SYNTACTICALLY AND SEMANTICALLY ANNOTATEDDATA,” and entity recognition and disambiguation technology is describedin detail in U.S. patent application Ser. No. 12/288,158, filed Oct. 15,2008, and entitled “NLP-BASED ENTITY RECOGNITION AND DISAMBIGUATION,”both of which are incorporated herein by reference in their entirety.The use of relationship searching, enables the CRS 200 to establishsecond order (or greater order) relationships between entities and tostore such information.

For example, given a sentence such as “Sean Connery starred inGoldfinger,” the identifier 212 may identify “Sean Connery” as thesentence subject, “starred” as the sentence verb, and “Goldfinger” asthe sentence object. The identified subjects and objects are then addedas disambiguated entities to the entity store 217 b. In the aboveexample, “Sean Connery” and “Goldfinger” would be added to the entitystore 217 b. The identified verbs can then be used to definerelationships between the identified entities. These definedrelationships (e.g., stored as subject-verb-object or SAO triplets, orotherwise) are then stored in the relationship index 217 d. In the aboveexample, a representation of the fact that the actor Sean Connerystarred in the film Goldfinger would be added to the relationship index217 d. In some embodiments, the process of identifying entities may beat least in part manual. For example, entities may be provisionallyidentified by the identifier 212, and then submitted to humans forediting, finalization, review, and/or approval.

The entity and relationship identifier 212 may determine various kindsof information about entities and relationships. In one embodiment, theidentifier 212 determines facets, which include finely grainedcharacteristics of entities, such as entity types, roles, qualities,functions, and the like. For example, the entity Sean Connery may havevarious associated facets, including that of actor, producer, knight,and Scotsman. The facet information for entities may be also stored inthe entity store 217 b.

A user 202 creates a collection by specifying or otherwise identifyingone or more entities. In some embodiments, the user 202 interacts with aWeb site or other type of user interface to create a collection byproviding a collection name and indicating one or more entities (orother things such as facets or keywords) to be included in thecollection. Collections specified by the user 202 are stored in thecollection store 217 c. Techniques for collection creation andcollection management are discussed further with reference to FIGS.3A-3I, below.

Given a collection of entities, the article recommender 214 determines,with reference to information contained in the data store 217, one ormore articles that are related to the collection. As discussed above,determining the one or more articles may include finding one or morearticles that cover or otherwise reference entities from the collection.The article recommender 214 then provides the determined articles to theuser 202, such as via a Web site, mobile device, or other userinterface. Techniques for article recommendation are discussed furtherwith reference to FIGS. 6, 7, 9, and 10, below.

Given a collection of entities, the entity recommender 213 determines,with reference to information contained in the data store 217, one ormore entities that are related to the collection. As discussed above,determining the one or more entities may include finding one or moreentities that are related to the entities of the collection, based onfactors such as whether the determined entities share characteristics(e.g. facets) with the entities of the collection, appear inrelationships with entities of the collection, and/or appear in contextwith entities of the collection. Techniques for entity recommendationare discussed further with reference to FIGS. 6, 8, 9, and 11, below.

The described techniques herein are not limited to the specificarchitecture shown in FIG. 2. For example, in some embodiments, contentingestion and entity and relationship identification may be performed byanother (possibly external or remote) system or component. In otherembodiments, the CRS 200 may not interact directly with users, butrather provide user interface components (e.g., collection widgets) thatmay be embedded or otherwise incorporated in third-party applications orsystems, such as Web sites, smart phones, desktop systems, and the like.

In addition, although the described techniques for contentrecommendation are illustrated primarily with respect to articlerecommendation and entity recommendation, other types of content arecontemplated. For example, other embodiments may utilize at least someof the described techniques to perform or facilitate the recommendationof other types of content, including advertisements, audio (e.g.,music), video, images, and the like.

FIGS. 3A-3I illustrate example screen displays provided by an exampleembodiment of a content recommendation system. More specifically, FIGS.3A-3I depict a running example, in which a user interacts with a Website operated by the content recommendation system to create acollection of entities and then obtain information about articles andentities that are related to the collection.

FIG. 3A shows a home page provided by the content recommendation system.In particular, FIG. 3A illustrates a Web browser 300 displaying a screen301 provided to a user by a content recommendation system. The screen301 is a home or main page provided by the content recommendationsystem, such as may be provided to the user when he or she firstaccesses the content recommendation system. Various examples ofdifferent user interfaces for accessing a content recommendation systemare described in detail in U.S. patent application Ser. No. 12/288,349,filed Oct. 16, 2008, and entitled “NLP-BASED CONTENT RECOMMENDER,” whichis incorporated herein by reference in its entirety The screen 301includes a search tool 302, a collections information area 303, and anarticles area 304. As will be described further below, the user can usethe search tool 302 to initiate searches for information about content(e.g., articles, entities) known to the content recommendation system.The collections information area 303 provides information aboutcollections that are associated with (e.g., created by) the user. At thepoint in time the current example is illustrated, the user has not yetdefined any collections. The articles area 304 provides informationabout articles known to the content recommendation system, possiblyarranged by categories, such as popular, politics, business,entertainment, sports, technology, and the like.

FIG. 3B illustrates initiation of a search via the contentrecommendation system. In particular, FIG. 3B shows the Web browser 300displaying screen 301 during the initiation of a search by the user. Asthe user enters text into the search tool 302, a drop down menu 310 isgenerated and displayed. The drop down menu 310 displays multipleindications of entities that match a string (e.g., “sean con”) enteredby the user, such that the user can select one of the entities as thesubject of the search. In the present example, the user selects anentity named “Sean Connery.”

FIG. 3C shows an entity information page provided as a result of thesearch initiated in FIG. 3B. In particular, FIG. 3C illustrates the Webbrowser 300 displaying a screen 320 provided in response to a search forthe entity named “Sean Connery.” The screen 320 is an entity informationpage that includes an entity relationship graph 321, a related articlesarea 322, a trend area 323, and a collection creation control 324(labeled “Follow”). The relationship graph 321 graphically presentsrelationships between Sean Connery and one or more other entities. Therelated articles area 322 displays indications of articles thatreference Sean Connery. The trend area 323 provides an indication of howfrequently Sean Connery as been mentioned over a particular time period(e.g., 30 days). In the present example, the user selects control 324(e.g., the “Follow” link) to initiate the creation of a new collection.

FIG. 3D depicts collection creation. In particular, FIG. 3D illustratesscreen 320 with a new collection dialog 330 displayed in response toselection of the collection creation control 324. Using the newcollection dialog 330, the user can name a new collection (e.g.,“Collection1”) that includes the entity Sean Connery.

FIG. 3E illustrates a response to the creation of a new collection. Inparticular, FIG. 3E illustrates screen 320 with a message 340 thatinforms the user that the entity Sean Connery is part of a collectionnamed Collection1. The message 340 includes a control (e.g., a link) 341that can be selected by the user to access a collection informationpage. In the present example, the user selects control 341.

FIG. 3F illustrates a collection information page. In particular, FIG.3F illustrates a screen 350 that is a collection information page forthe collection named Collection1, created as described above. The screen350 includes a collection membership area 351, a related entity area352, a related articles area 353, and an add entity control 354. Thecollection membership area 351 displays information, such as names andfacets, about the entities of the current collection. In the presentexample, the collection membership area 351 indicates that Collection1includes a single entity named “Sean Connery,” who is an actor (e.g.,has the facet: actor). The related entity area 352 displays information(e.g., names and facets) about entities that are related to the entitiesof Collection1. The related articles area 352 displays information(e.g., titles, text snippets, entities referenced) about articles thatare related to the entities of Collection1. The entity control area 354can be selected by the user to add another entity to the currentcollection, as will be described below in FIG. 3G.

FIG. 3G illustrates the addition of an entity to an existing collection.In particular, FIG. 3G illustrates the collection information page ofscreen 350 during the addition of an entity to Collection1. Here, theuser has selected the add entity control 354 (FIG. 3F) and in response,an entity addition dialog 360 is displayed. The entity addition dialog360 includes a search tool with a drop down menu that is populated withentities matching a string (e.g., “Roger Mo”) entered by the user. Inthe present example, the user selects the entity named “Roger Moore.”

FIG. 3H shows a response to the addition of an entity to a collection.In particular FIG. 3H illustrates the screen 350 and collectioninformation area 351 introduced in FIG. 3F. In the present example,collection information area 351 has been modified to include the entitynamed “Roger Moore,” reflecting the addition to Collection1 performed asdescribed with reference to FIG. 3G.

FIG. 3I shows an alternative mechanism for adding an entity to acollection. In particular, FIG. 3I illustrates an entity addition dialog370 that is configured to present multiple entities that are related tothe entities of Collection1. The related multiple entities are presentedas suggested additions that can be selected by the user without the needto type the name of a particular entity. Such a user interface controlcan be provided in addition to, or instead of, the entity additiondialog 360 described with reference to FIG. 3G.

FIGS. 4A-4D illustrate additional example screen displays provided by anexample embodiment of a content recommendation system. Morespecifically, FIGS. 4A-4D depict various additional or alternativeWeb-based user interface techniques for creating, utilizing, andmanaging collections.

FIG. 4A shows a widget gallery provided by a content recommendationsystem. In particular, FIG. 4A illustrates a Web browser 400 displayinga screen 401 provided to a user by a content recommendation system. Thescreen 401 is a widget gallery provided by the content recommendationsystem. Screen 401 includes a control (e.g., a link) 402 that can beselected by a user to create a collections widget. A collections widgetis a code module that can be included in an application (e.g., a 3rdparty application) and that is configured to provide contentrecommendations based on an associated collection.

FIG. 4B illustrates the creation of a collections widget. In particular,FIG. 4B illustrates a screen 410 displayed in response to a userselection of control 402. The screen 410 includes a collections widgetcreation tool 411 that includes a search tool and drop down menu thatcan be selected by the user to specify one or more entities as membersof a new collection. In the present example, the user is creating acollections widget about soccer players, including Lionel Messi andDiego Maradona.

FIG. 4C shows a collections widget preview. In particular, FIG. 4Cillustrates a screen 420 that is a collections widget preview pagedisplayed in response to the creation of a collections widget asdescribed with reference to FIG. 4B. The screen 420 includes a widgetpreview area 421 and a widget code area 422. The preview area 421displays a preview of the created widget, as it would appear if it wereembedded in a third-party application, such as a Web page, blog, or thelike. The code area 422 provides instructions (e.g., JavaScript code)that can be copied by a user into some other application. The providedinstructions are configured, when executed, to display the collectionswidget previewed in the preview area 421.

FIG. 4D shows a collections gallery. In particular, FIG. 4D illustratesa screen 430 that includes a collections gallery 431 that displaysinformation about collections created by various users of the contentrecommendation system. In this manner, users of the contentrecommendation system can share their own collections and/or learn aboutcollections created by other users. In the illustrated embodiment, acollection created by a user is by default public and thus accessible byall other users of the content recommendation system. Other embodimentsprovide various types of access control. In one embodiment, a user canindicate or define specific users and/or groups of users that can accesssome or all of the user's collections. In addition, a user can specifydefault access control policies that are to be applied to theircollections, such as private, public, group accessible, and the like.

Although the user interface techniques of FIGS. 3A-3I and 4A-4D havebeen described primarily with reference to Web-based technologies, thedescribed techniques are equally applicable in other contexts. Forexample, a collections widget may be implemented as one or more codeobjects (e.g., a Java object) that can be embedded within a client-sidedesktop or mobile application, and configured to interact with thecontent recommendation system to obtain and display contentrecommendations based on an associated collection.

FIG. 5 is an example block diagram of an example computing system forimplementing a content recommendation system according to an exampleembodiment. In particular, FIG. 5 shows a computing system 500 that maybe utilized to implement a content recommendation system 510.

Note that one or more general purpose or special purpose computingsystems/devices may be used to implement the content recommendationsystem 510. In addition, the computing system 500 may comprise one ormore distinct computing systems/devices and may span distributedlocations. Furthermore, each block shown may represent one or more suchblocks as appropriate to a specific embodiment or may be combined withother blocks. Also, the content recommendation system 510 may beimplemented in software, hardware, firmware, or in some combination toachieve the capabilities described herein.

In the embodiment shown, computing system 500 comprises a computermemory (“memory”) 501, a display 502, one or more Central ProcessingUnits (“CPU”) 504, Input/Output devices 504 (e.g., keyboard, mouse, CRTor LCD display, and the like), other computer-readable media 505, andnetwork connections 506. The content recommendation system 510 is shownresiding in memory 501. In other embodiments, some portion of thecontents, some or all of the components of the content recommendationsystem 510 may be stored on and/or transmitted over the othercomputer-readable media 505. The components of the contentrecommendation system 510 preferably execute on one or more CPUs 503 andrecommend content based on entity collections, as described herein.Other code or programs 530 (e.g., an administrative interface, a Webserver, and the like) and potentially other data repositories, such asdata repository 520, also reside in the memory 501, and preferablyexecute on one or more CPUs 503. Of note, one or more of the componentsin FIG. 5 may not be present in any specific implementation. Forexample, some embodiments may not provide other computer readable media505 or a display 502.

In a typical embodiment, the content recommendation system 510 includesa content ingester 511, an entity and relationship identifier 512, anentity recommender 513, an article recommender 514, a user interfacemanager 515, a recommender application program interface (“API”) 516,and a data store 517. The content ingester 511, entity and relationshipidentifier 512, user interface manager 515, and recommender API 516 aredrawn in dashed lines to emphasize that in other embodiments, functionsperformed by one or more of these components may be performed externallyto the content recommendation system 510.

The content ingester 511 performs functions such as those described withreference to the content ingester 211 of FIG. 2. The ingester 511obtains content items, such as Web pages, Blog postings, videos, audiofiles, and the like from various content sources 555 via network 550,and stores information about the obtained content items in the datastore 517, for use by other components, such as the entity andrelationship identifier 512, the entity recommender 513, and/or thearticle recommender 514.

The entity and relationship identifier 512 performs functions such asthose described with reference to the entity and relationship identifier212 of FIG. 2. The identifier 512 processes the information aboutcontent items stored in the data store 517 in order to identify entitiesand relationships between them. Identified entities and relationshipsare stored in the data store 517, for use by other components, such asthe recommenders 513 and 514.

The UI manager 515 provides a view and a controller that facilitate userinteraction with the content recommendation system 510 and its variouscomponents. For example, the UI manager 515 may provide interactiveaccess to the content recommendation system 510, such that users cancreate collections, manage/edit collections, share collections, obtainrecommendations based on collections, and the like. In some embodiments,access to the functionality of the UI manager 515 may be provided via aWeb server, possibly executing as one of the other programs 530. In suchembodiments, a user operating a Web browser executing on one of theclient devices 560 can interact with the content recommendation system510 via the UI manager 515.

The entity recommender 513 and article recommender 514 perform functionssuch as those described with reference to the entity recommender 213 andarticle recommender 214, respectively, of FIG. 2. The recommenders 513and 514 receive, possibly via the UI manager 515, an indication of acollection (e.g., stored in the data store 517). In response, therecommenders 513 and 514 respectively determine one or more contentitems (e.g., entities and/or articles) that are related to entities ofthe indicated collection. The determined content items can be cached(e.g., stored in the data store 517) for later use and/or provideddirectly in response to a received request, such as may be generated bythe UI manager 515.

In one embodiment, the recommenders 513 and 514 operate synchronously inan on-demand manner, in that they perform their functions in response toa received request, such as in response to a user interface eventprocessed by the UI manager 515. In another embodiment, the recommenders513 and 514 operate asynchronously, in that they automatically determinerelated content items for one or more collections. For example, therecommenders 513 and 515 may automatically execute from time to time(e.g., once per hour, once per day) in order to generate bulk contentrecommendations for some or all collections stored in the data store517. The recommenders 513 and 514 may execute upon the occurrence ofother types of conditions, such as when a new collection is created,when new content items are indexed, and the like.

The API 516 provides programmatic access to one or more functions of thecontent recommendation system 510. For example, the API 516 may providea programmatic interface to one or more functions of the contentrecommendation system 510 that may be invoked by one of the otherprograms 530 or some other module. In this manner, the API 516facilitates the development of third-party software, such as userinterfaces, plug-ins, news feeds, adapters (e.g., for integratingfunctions of the content recommendation system 510 into Webapplications), and the like. In addition, the API 516 may be in at leastsome embodiments invoked or otherwise accessed via remote entities, suchas one of the third-party applications 565, to access various functionsof the content recommendation system 510. For example, a third-partyapplication may create a collection and/or receive recommendations basedon a collection from the content recommendation system 510 via the API516. The API 516 may also be configured to provide collections widgets(e.g., code modules) that can be integrated into third-partyapplications and that are configured to interact with the contentrecommendation system 510 to make at least some of the describedfunctionality available within the context of other applications.

The data store 517 is used by the other modules of the contentrecommendation system 510 to store and/or communicate information. Asdiscussed above, components 511-516 use the data store 517 to recordvarious types of information, including content, information aboutstored content including entities and relationships, information aboutcollections, user information, and the like. Although the components511-516 are described as communicating primarily through the data store517, other communication mechanisms are contemplated, including messagepassing, function calls, pipes, sockets, shared memory, and the like.

The content recommendation system 510 interacts via the network 550 withcontent sources 555, third-party applications 565, and client computingdevices 560. The network 550 may be any combination of media (e.g.,twisted pair, coaxial, fiber optic, radio frequency), hardware (e.g.,routers, switches, repeaters, transceivers), and protocols (e.g.,TCP/IP, UDP, Ethernet, Wi-Fi, WiMAX) that facilitate communicationbetween remotely situated humans and/or devices. The client computingdevices 560 include desktop computing systems, notebook computers,mobile phones, smart phones, personal digital assistants, tabletcomputers, and the like.

Other or additional functions and/or data are contemplated. For example,in some embodiments, the content recommendation system 510 includesadditional content recommendation components that are specialized toother types of content, such as for video, quotations, images, audio,advertisements, product information, and the like.

In an example embodiment, components/modules of the contentrecommendation system 510 are implemented using standard programmingtechniques. For example, the content recommendation system 510 may beimplemented as a “native” executable running on the CPU 503, along withone or more static or dynamic libraries. In other embodiments, thecontent recommendation system 510 may be implemented as instructionsprocessed by a virtual machine that executes as one of the otherprograms 530. In general, a range of programming languages known in theart may be employed for implementing such example embodiments, includingrepresentative implementations of various programming languageparadigms, including but not limited to, object-oriented (e.g., Java,C++, C#, Visual Basic.NET, Smalltalk, and the like), functional (e.g.,ML, Lisp, Scheme, and the like), procedural (e.g., C, Pascal, Ada,Modula, and the like), scripting (e.g., Perl, Ruby, Python, JavaScript,VBScript, and the like), and declarative (e.g., SQL, Prolog, and thelike).

The embodiments described above may also use either well-known orproprietary synchronous or asynchronous client-server computingtechniques. Also, the various components may be implemented using moremonolithic programming techniques, for example, as an executable runningon a single CPU computer system, or alternatively decomposed using avariety of structuring techniques known in the art, including but notlimited to, multiprogramming, multithreading, client-server, orpeer-to-peer, running on one or more computer systems each having one ormore CPUs. Some embodiments may execute concurrently and asynchronously,and communicate using message passing techniques. Equivalent synchronousembodiments are also supported. Also, other functions could beimplemented and/or performed by each component/module, and in differentorders, and by different components/modules, yet still achieve thedescribed functions.

In addition, programming interfaces to the data stored as part of thecontent recommendation system 510, such as in the data store 517, can beavailable by standard mechanisms such as through C, C++, C#, and JavaAPIs; libraries for accessing files, databases, or other datarepositories; through scripting languages such as XML; or through Webservers, FTP servers, or other types of servers providing access tostored data. The data store 517 may be implemented as one or moredatabase systems, file systems, or any other technique for storing suchinformation, or any combination of the above, including implementationsusing distributed computing techniques.

Different configurations and locations of programs and data arecontemplated for use with techniques of described herein. A variety ofdistributed computing techniques are appropriate for implementing thecomponents of the illustrated embodiments in a distributed mannerincluding but not limited to TCP/IP sockets, RPC, RMI, HTTP, WebServices (XML-RPC, JAX-RPC, SOAP, and the like). Other variations arepossible. Also, other functionality could be provided by eachcomponent/module, or existing functionality could be distributed amongstthe components/modules in different ways, yet still achieve thefunctions described herein.

Furthermore, in some embodiments, some or all of the components of thecontent recommendation system 510 may be implemented or provided inother manners, such as at least partially in firmware and/or hardware,including, but not limited to one or more application-specificintegrated circuits (“ASICs”), standard integrated circuits, controllersexecuting appropriate instructions, and including microcontrollersand/or embedded controllers, field-programmable gate arrays (“FPGAs”),complex programmable logic devices (“CPLDs”), and the like. Some or allof the system components and/or data structures may also be stored ascontents (e.g., as executable or other machine-readable softwareinstructions or structured data) on a computer-readable medium (e.g., asa hard disk; a memory; a computer network or cellular wireless networkor other data transmission medium; or a portable media article to beread by an appropriate drive or via an appropriate connection, such as aDVD or flash memory device) so as to enable or configure thecomputer-readable medium and/or one or more associated computing systemsor devices to execute or otherwise use or provide the contents toperform at least some of the described techniques. Some or all of thesystem components and data structures may also be stored as data signals(e.g., by being encoded as part of a carrier wave or included as part ofan analog or digital propagated signal) on a variety ofcomputer-readable transmission mediums, which are then transmitted,including across wireless-based and wired/cable-based mediums, and maytake a variety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Some or allof the system components and data structures may also be stored in oneor more non-transitory computer-readable storage mediums. Such computerprogram products may also take other forms in other embodiments.Accordingly, embodiments of this disclosure may be practiced with othercomputer system configurations.

FIG. 6 is an example flow diagram of a content recommendation processperformed by an example content recommendation system according to anexample embodiment. In particular, FIG. 6 illustrates a process that maybe implemented by, for example, one or more elements of the contentrecommendation system 200, such as the entity recommender 213 and/or thearticle recommender 214, as described with reference to FIG. 2. Theprocess recommends content items based on entities of a collection.

The illustrated process begins at block 602, where it defines acollection of entities. Defining a collection of entities may includereceiving indications of one or more entities, such as may be providedby an interactive user interface operated by a user. An indication of anentity may be or include a unique identifier of an entity. In othercases, the indication of the entity may be ambiguous, in that it mayrefer to more than one entity. In still other cases, a class or group ofentities may be specified by a facet. For example, the facet “Actor”could be used to indicate all entities that are known to be actors. Inother cases, one or more keyterms that include one or more words orphrases may be used to indicate one or more entities. For example, thekeyterms “soccer player” may be used to indicate all entities that haveor are associated with textual descriptions that match, or approximatelymatch, the keyterms “soccer player.” Defining a collection may furtherinclude storing received entity indications in a data structure or otherdata arrangement, such as a file, database, or the like.

At block 604, the process determines content items, such as articles orentities, related to entities of the collection. In one embodiment,determining related content items includes locating content items thatcover, include, or otherwise reference one or more entities of thecollection. For example, determining related articles may includelocating articles that each reference at least one of the entities ofthe collection. In another embodiment, determining related content itemsincludes determining content items that are in a relationship with oneor more of the entities of the collection. For example, determiningrelated entities may include determining one or more entities that areeach in a relationship with at least one of the entities of thecollection. In a further embodiment, determining related content itemsincludes locating content items that have characteristics that match, orare in some way similar to, characteristics of one or more entities ofthe collection. For example, determining related entities may includedetermining entities that share at least one facet with at least one ofthe entities of the collection. Determining related content items mayalso include ordering and/or ranking the content items based on variousfactors, such as relevance, credibility, recency, and the like.Determining related articles and entities are discussed further withrespect to FIGS. 7 and 8, respectively.

At block 606, the process recommends the determined content items.Recommending the determined content items may include providing (e.g.,displaying, presenting, transmitting, storing, forwarding) indicationsof the content items, such as via a user interface screen, as describedabove with reference to FIGS. 1 and 3A-3I. In some embodiments,indications of the determined content items may be provided indirectly,such as via a user interface widget or other component that isincorporated into a third-party application. In other embodiments,recommending the determined content items includes storing the contentitems (or indications thereof) in a data store for later consumption oruse by another component.

Some embodiments perform one or more operations/aspects in addition to,or instead of, the ones described with reference to the process of FIG.6. For example, in one embodiment, the process also provides userinterface widgets configured to provide content recommendations in thecontext of a third-party application, such as those described withreference to FIGS. 4A-4D.

FIG. 7 is an example flow diagram of an article recommendation processperformed by an example content recommendation system according to anexample embodiment. In particular, FIG. 7 illustrates a process that maybe implemented by, for example, one or more elements of the contentrecommendation system 200, such as the article recommender 214, asdescribed with reference to FIG. 2. The process recommends articlesbased on entities of a collection.

The illustrated process begins at block 702, where it receives anindication of a collection including entities E1, E2, . . . , E3 andnumber of articles to recommend (N).

At block 704, the process initializes an article set A to be the emptyset (A={ }). The process also initializes an entity set E to include allof the entities of the collection (E={E1, E2, . . . , E3}). The processfurther initializes a threshold T=N/|E|, or in other words initializes Tto be the quotient of the number of articles to recommend (N) over thecardinality of the set E (|E|). The process attempts to find, for eachentity, at least T articles that reference that entity.

In blocks 706-714, the process performs a loop in which it retrievesarticles that are related to one or more of the entities in the set E,and removes entities from E when sufficient articles covering theremoved entities have been retrieved, until E is empty, or some otherstopping condition occurs.

At block 706, the process determines an ordered list of relevantarticles that each reference at least one entity in E. To determine theordered list of articles, the process performs a query that is a BooleanOR of all of the entities in the set E. For example, if the set Eincludes entities E1, E2, and E3, the resulting query would be id(E1) ORid(E2) OR id(E3), where the operator id(X) returns a unique identifierfor an entity X. If any of the elements of the set E refer to two ormore entities (e.g., are ambiguous), facets, or keyterms, the resultingquery will incorporate such elements, such as by including the keyterms,facets, or other disambiguating information, as discussed further withrespect to FIG. 9, below.

The determined list of articles is ordered based on an informationretrieval weighting that is based at least in part on one or more of acredibility score, a recency score, a term frequency, total number ofterms in the article, length of the article, and presence of terms inthe title of the article. An example process for scoring an article forpurposes of ordering the list of articles is described with reference toFIG. 10, below.

At block 708, the process retrieves and adds the top N articles of thedetermined list of articles to the resultant set A. In some embodiments,determining the list of articles (as in block 706) is computationallyless expensive than actually retrieving the articles. For example,determining the list of articles may be performed by one or moredatabase queries that operate against indexed article information, andthus may execute relatively quickly as compared to the cost of actuallyretrieving article content, which is typically stored external to thedatabase and thus may have a correspondingly larger access and retrievaltime. Thus, as illustrated here, the process only retrieves the top Narticles indicated in the ordered list of articles, and adds them to theset A, thus adding the N most relevant articles (with respect to thequery of block 706) to the set A. In one example embodiment, N is 10.Other example embodiments may incorporate a different number of desiredarticles.

At block 710, the process determines whether there are not enougharticles (the number of articles in the determined list of articles isless than N), and if so, continues at block 716, otherwise at block 712.With this check, the process performs an early exit from the loop ofblocks 706-714 in cases when the total number of articles determined inblock 706 is less than or equal to the number of articles to recommend(N). It is appropriate to exit the loop of blocks 706-714 in this casebecause performing additional queries will not result in additionalarticles being returned. For example, if 10 articles have been requested(N=10) and the query of block 706 only returned four articles, theprocess would exit, because a subsequent query would not result in anyarticles other than those four articles being returned.

At block 712, the process, for each entity in the set E, counts thenumber of articles in the set A that reference the entity. If the countis greater than the threshold T, the process removes the entity from setE. In other words, when an entity has been sufficiently covered byarticles in the resultant set A, the entity will be removed from the setE, thereby reducing the size of the set E and moving the process onestep closer to its stopping condition of an empty set E. Removingentities from set E will cause the performance of a different query onsubsequent iterations of the loop of blocks 706-714, typically resultingin the determination of a list of articles that is different fromprevious query results in membership and/or order (due to theinformation retrieval weighting being different due to a differentquery).

At block 714, the process determines whether the set E is empty, and ifso, continues at block 716, otherwise continues the loop of blocks706-714. When the set of entities (E) is empty, then a sufficient numberof articles covering each of the entities of the collection have beenretrieved, and the loop terminates. In some embodiments, an additionalstopping condition is included to limit the maximum number of iterationsof the loop of blocks 706-714. For example, the test at block 714 mayinstead be to determine whether the set E is empty or whether the loopof blocks 706-714 has iterated a maximum number of times (e.g., fourtimes). Such a test bounds the running time of the process in cases oflarge collections (e.g., where the set E has many elements).

At block 716, the process sorts the resultant article set A based onarticle relevance. Article relevance may be based on various factors,including article recency (e.g., publishing date of each article) andentity coverage (e.g., the number of entities of the collectionreferenced in each article).

After block 716, the process returns an indication of the ordered set ofarticles A. Returning the set A may include returning a data structure(e.g., an XML record) that includes references to each of the articlesin set A. In other embodiments, the details of the set A are stored in adatabase record (or other persistent data store), and an identifier forthe database record is returned.

Some embodiments perform one or more operations/aspects in addition to,or instead of, the ones described with reference to the process of FIG.7. For example, although the process is described iteratively here, inother embodiments, the process is implemented recursively. Also, insteadof consecutively performing separate, different queries, otherembodiments may perform multiple different queries in parallel. Inaddition, other embodiments may perform queries that differ from thosediscussed above, including queries that connect one or more of theentities using different operators (e.g., logical AND, NOT, etc.).Techniques for article recommendation used in a specific exampleembodiment are described further below.

FIG. 8 is an example flow diagram of an entity recommendation processperformed by an example content recommendation system according to anexample embodiment. In particular, FIG. 8 illustrates a process that maybe implemented by, for example, one or more elements of the contentrecommendation system 200, such as the entity recommender 213, asdescribed with reference to FIG. 2. The process recommends entitiesbased on entities of a collection.

The illustrated process begins at block 802, where it receives anindication of a collection including multiple entities.

At block 804, the process determines a first set of entities that areeach related to one of the multiple entities. Determining the first setof entities includes searching for entities that appear in relationshipsexpressed in content items such as articles. As discussed above, thecontent recommendation system may include an entity and relationshipdata store that represents relationships between entities. This datastore can be searched for all entities that are related to at least oneof the entities of the collection. In one embodiment, the processinitiates multiple queries, where each query is directed to findingentities related to one of the entities of the collection. Morespecifically, each query asks for the set of entities that are one ofthe subject or object of a subject-verb-object relation in which the oneentity of the collection appears as the other of the subject or object.Typically, the process executes the multiple queries in parallel, inorder to efficiently locate a broad set of entities related to theentities of the collection. For example, given a collection includingSean Connery and Roger Moore, the process would issue two queries: afirst one requesting entities in relationships with Sean Connery and asecond one requesting entities in relationships with Roger Moore.Examples of these queries and corresponding example search results usingan example embodiment are described with reference to FIGS. 11A and 11B,below.

At block 806, the process determines shared characteristics of the firstset of entities, including common facets and common keyterms.Determining the shared characteristics of the first set of entitiesincludes iterating over the relationships returned by the multiplequeries performed at block 804, and counting the related entities (notthe collection entities) occurring in the relationships, facets assignedto the each of the entities of the relationships, and keyterms occurringin the relationships. For example, given a collection that includesRoger Moore and Sean Connery, particular entities may commonly appear inrelationships with those entities, such as James Bond. Also, facets thatcould be common to the example collection include actor and knight.

Determining the shared characteristics (e.g., facets) may also includeranking the shared characteristics. For example, facets may be ranked byfrequency counts, such that more commonly appearing facets occur priorto less commonly appearing facets. At the end of block 806, the processhas determined a ranked list of the main facets shared by the entitiesin the collection. For example, given a collection including Roger Mooreand Sean Connery, shared facets could include actor and knight.

In addition, keyterms may be ranked based on measures of documentfrequency, such as based on the product of term document frequency andinverse document frequency, where term document frequency is the numberof times a term appears in a given document, and inverse documentfrequency is the logarithm of the quotient of the total number ofdocuments and the number of documents containing the term. In thismanner, the keyterms that are most relevant and specific to the givencollection can be identified. For example, given a collection includingRoger Moore and Sean Connery, example keyterms could include “007” and“bond actor.”

At block 808, the process determines a second set of entities that eachappear in an article in a relationship with one of the shared/commonfacets identified in block 806 and in context with one of the commonkeyterms identified in block 806. This block results in finding otherentities that are potentially similar to the first set, through theircommon facets and/or keywords. Determining the second set of entitiesincludes performing a query that searches for entities that are in arelationship with an entity having at least one of the common facets(e.g., find entities that are in a relationship with an actor), therelationship appearing in an article in context with at least one of thekeyterms (e.g., the keyterm “007”), where the common facets and keytermsare those identified at block 806, above. One entity appears in contextwith another entity or keyterm if it appears in an article within apredefined window of text (e.g., 50 characters, three sentences, sameparagraph) around the other entity. For example, given a collectionincluding Roger Moore and Sean Connery, the process finds a second setof entities that are in a relationship with a common facet, such asactor, the relationships appearing in an article in context with acommon keyterm, such as “007.” Such entities could include, for example,James Bond, Daniel Craig, Pierce Brosnan, and the like. After performingthe above queries, the process iterates through the returnedrelationships and collects entities and their frequency counts. Anexample of the type of query performed at block 808 and correspondingexample search results using an example embodiment are described withreference to FIG. 11C, below. Additional examples of the kinds ofqueries performed at block 808 are described further below.

At block 810, the process determines a third set of entities that eachappear in an article in a relationship with one of the multiple entitiesand in context with another of the multiple entities. Determining thethird set of entities includes performing multiple queries to captureinter connections between the entities in the collection. Performing themultiple queries includes, for each entity of the collection, performinga query that searches for entities that are in a relationship with theentity of the collection, the relationship appearing in an article incontext with one of the other entities of the collection. Again, thesemultiple queries can be performed in parallel. For example, given acollection including Roger Moore and Sean Connery, the process hereperforms two queries: first, a query that searches for entities relatedto Roger Moore and appearing in context with Sean Connery, and second, aquery that searches for entities related to Sean Connery and appearingin context with Roger Moore. Examples of these queries and correspondingexample search results using an example embodiment are described withreference to FIGS. 11D and 11E, below. After performing the multiplequeries, the process again iterates through the returned relationshipsand collects entities and their frequency counts.

At block 812, the process orders a combination of the first, second, andthird sets. In some embodiments, ordering the combination of the threesets is based on entity frequency counts and whether entities of thecombination share one or more of the common facets identified in block806. For example, entities of the combination may be ordered by theirfrequency score (e.g., more frequently occurring entities occurringearlier in the combination) multiplied by a facet score, which is ameasure of how many entities in the original collection also have aparticular facet. For example, given a collection containing Roger Mooreand Sean Connery, which both have the actor facet, the process may findtwo related entities, Daniel Craig and James Bond. As between DanielCraig and James Bond, Daniel Craig would have a higher facet score thanJames Bond, because Daniel Craig also has the actor facet.

After block 812, the process returns the ordered combination ofentities. Returning the ordered combination may include returning a datastructure (e.g., an XML record) that includes references to each entityin the combination. In other embodiments, the details of the orderedcombination are stored in a database record (or other persistent datastore), and an identifier for the database record is returned.

Some embodiments perform one or more operations/aspects in addition to,or instead of, the ones described with reference to the process of FIG.8. For example, in one embodiment, the process constrains its executiontime by considering a predetermined number of entities, facets,keyterms, and the like. For example, at block 806, the process may electto iterate over a predetermined number of relationships (e.g., the top10) returned by the multiple queries returned at block 804, where therelationships returned by the multiple queries are ranked in order ofrelevance. As another example, at block 808 the process may generate aquery that includes only a predetermined number of the most commonfacets and/or keyterms. Techniques for entity recommendation used in aspecific example embodiment are described further below.

FIG. 9 is an example flow diagram of a query generation processperformed by an example content recommendation system according to anexample embodiment. In particular, FIG. 9 illustrates a process that maybe implemented by, for example, one or more elements of the contentrecommendation system 200, as described with reference to FIG. 2. Theprocess generates a query or a portion of a query that can be used todetermine (e.g., search) for entities, relationships, and/or articles,as described with reference to FIGS. 7 and/or 8, above.

The illustrated process begins at block 902, where it receives anindication of an element of a collection. As noted, in some embodiments,collections include various types of indications of singular or pluralentities. Thus, the collection element may be or include a name of oneor more entities (e.g., the string “Sean Connery,” “James Bond,”“Michael Jackson”), a facet of one or more entities (e.g., actor,politician, book, film), or a keyterm (e.g., one or more wordsdescribing one or more entities, including “soccer player,” “goodactor,” and “prescription drugs”).

At block 904, the process determines whether the element refers to aknown and disambiguated entity (“KDE”), and if so, continues at block906, otherwise at block 908. The element refers to a KDE if it refers toan entity that has been uniquely identified and/or indexed by thecontent recommendation system. If this is the case, then the entity isassociated with a unique identifier (e.g., in the entity data store 217b) that can be used within a query.

At block 906, the process constructs a query that includes a uniqueidentifier associated with the entity. As noted, a KDE is associatedwith a unique identifier that can be used in a query. Thus, constructingthe query includes obtaining (e.g., looking up in the entity data store217 b) the unique identifier and generating a query (or portion thereof)that includes the identifier. Then, the process continues at block 920.

At block 908, the process determines whether the element refers to aknown but not-disambiguated entity (“KNDE”), and if so, continues atblock 910, otherwise at block 912. The element refers to a KNDE if itrefers to one or more entities that are known to the contentrecommendation system but have not yet been indexed or otherwiseassigned a unique identifier.

At block 910, the process constructs a query that includes the name ofthe entity and optionally disambiguating terms. Constructing the queryincludes determining whether the entity needs to be disambiguated. Insome cases, such as when the entity name is at least substantiallyunique (e.g., as determined based on inverse document frequency, wherenames with a higher inverse document frequency are typically moreunique), there is no need to disambiguate the entity, and using theentity name alone may suffice. Other cases where disambiguation may notbe needed include entities that are emerging popular entities that arecurrently widely covered in the news and thus typically do not needdisambiguation. Emerging popular entities can be determined withreference to various sources, such as Wikipedia page view counts,Wikipedia in-bound link counts, popular queries on major search engines,trending topics on social networks (e.g., Twitter), and the like. If theentity does not need to be disambiguated, the constructed query willtypically include just the entity name.

Cases where the entity needs to be disambiguated may include entitynames that refer to multiple entities, entity names that are commonnames (e.g., “Will Smith”), entities that are not emerging popularentities, or the like. When the entity needs to be disambiguated, theconstructed query includes the entity name and one or moredisambiguating terms. The disambiguating terms may be based on theentity's facets and/or related entities. For example, to query WillSmith the football player (as opposed to Will Smith the actor), aconstructed query could include the string “Will Smith” and variouskeyterms that express facets (e.g., “football player”) and/or entities(e.g., “New Orleans Saints,” “Super Bowl”) that are associated withand/or related to the entity Will Smith the football player. Afterconstructing the query, the process continues at block 920.

At block 912, the process determines whether the element is a keyterm,and if so, continues at block 914, otherwise at block 916. At block 914,the process constructs a query that includes all or some of the keyterm.In some embodiments, variations and/or stems of the keyterms may beincluded in the query. Then, the process continues at block 920.

At block 916, the process determines whether the element is a facet, andif so, continues at block 918, otherwise at block 922. In someembodiments, a facet may be identified by particular syntax, such assquare brackets (e.g., [actor], [football_player], [athlete]). At block918, the process constructs a query that includes a facet token. A facettoken identifies the facet such that articles that cover or otherwiserefer to the facet can be efficiently obtained. In some embodiments,articles are annotated with one or more facet tokens prior to indexing,such that articles can searched based on facet coverage. The one or morefacet tokens that annotate an article may be limited to a predeterminednumber of facets, such as those associated with the primary (e.g., mostrelevant or important) entities referenced in an article. In thismanner, the content recommendation system is configured to returnarticles that are substantially related to queried facet. Then, theprocess continues at block 920.

At block 922, the process provides an indication of an unknown elementtype. At this point, the process has been unable to match any of theknown element types, and returns an error or other indication (e.g.,raises an exception) that the collection element is not of a recognizedtype.

At block 920, the process provides the constructed query. Providing theconstructed query includes returning all or part of a query string thatcan be used by another process (e.g., the article recommendation processof FIG. 7) to search for content items such as articles or entities.

After blocks 920 and 922, the process returns. Some embodiments performone or more operations/aspects in addition to, or instead of, the onesdescribed with reference to the process of FIG. 9. Also, in someembodiments, not all of the discussed collection element types aresupported or required in every embodiment. For example, the process mayhandle only known disambiguated entities (KDEs) and known butnot-disambiguated entities (KNDEs).

FIG. 10 is an example flow diagram of an article scoring processperformed by an example content recommendation system according to anexample embodiment. In particular, FIG. 10 illustrates a process thatmay be implemented by, for example, one or more elements of the contentrecommendation system 200, as described with reference to FIG. 2. Theprocess scores an article (e.g., determines a weight or rank for anarticle) for purposes of ordering multiple articles during articlerecommendation, as described with reference to FIG. 7, above.

The illustrated process begins at block 1002, where it receives anindication of an article and a query.

At block 1004, the process determines a relevance score based on one ormore query terms. Determining the relevance score includes calculating ascore for the article with respect the query, based on one or moreinformation retrieval relevance metrics. One such metric is the productof term document frequency and inverse document frequency, where termdocument frequency is the number of times a term from the query appearsin the article, and inverse document frequency is the logarithm of thequotient of the total number of documents in a corpus (e.g., thedocument store 217 a) of the content recommendation system and thenumber of documents in the corpus that contain the term.

At block 1006, the process determines a credibility score for thearticle. In some embodiments, the article credibility score can bedetermined based on properties of the source of the article. Forexample, if the source of the article is a Website that receives a highvolume of traffic and/or is ranked highly by one or more search engines(e.g., via Google's PageRank or similar metrics), the article may beconsidered more credible than an article having a source Website withminimal traffic or low search engine rank.

At block 1008, the process determines a recency score for the article.In one embodiment, the recency score is based on the difference betweenthe article date (e.g., publication date, last edit date) and the dateof article ingestion by the content recommendation system, such thatarticles having a smaller such difference will have a greater recencyscore. In other embodiments, recency score may be based on individualfactors, such as ingestion date or publication date, taken alone.

At block 1010, the process determines an overall score based on therelevance score, credibility score, and recency score. In oneembodiment, the overall score is computed by multiplying the relevancescore, credibility score, and recency score together, along withappropriate scaling or normalization factors.

After block 1010, the process returns. Some embodiments perform one ormore operations/aspects in addition to, or instead of, the onesdescribed with reference to the process of FIG. 10. For example, in someembodiments, the process does not compute its own article relevancescore. Instead, such a score is computed by an underlying documentsearch and retrieval system, and the process instead calculates anarticle boosting score based on the product of the credibility score andrecency score. The document boosting score is then passed along to theunderlying document search and retrieval system for purposes of rankingsearch results. One such embodiment utilizes the Apache Lucene documentsearch library/system, as described in more detail below.

FIGS. 11A-11E illustrate example screen displays generated by queriesexecuted for content recommendation provided by an example embodiment ofa content recommendation system. In particular, FIGS. 11A-11E illustratesearch results provided by an example content recommendation system inresponse to received entity relationship queries, such as may begenerated by the entity recommendation process of FIG. 8, above. Forillustrative purposes, the entities and relationships of FIGS. 11A-11Eare presented and described in the context of a human-readable userinterface, whereas an automatic process such as that of FIG. 8 wouldprogrammatically access the same or similar information via anapplication program interface or other mechanism.

FIG. 11A shows a screen 1100 (e.g., a Web page) that displays searchresults returned in response to a first search for all entities relatedto a specified entity. The screen 1100 includes a search tool used tospecify a query 1101 and search results 1102. The search results 1102include multiple relationship records, e.g., records 1106, that eachdescribe one relationship. In this case, the query 1101 is of the form“Sean Connery” < > * < > *, which requests all relationships thatinclude the entity Sean Connery, using a relationship search querylanguage, such as “EQL” provided by Evri Inc. In response, the contentrecommendation system has returned multiple relationship records,including records 1106. Each of the records includes indicators of asubject 1103 (e.g., Chris Matthews), action 1104 (e.g., acts like), andobject 1105 (e.g., Sean Connery), as well as an expansion control 1107.In response to a user selection of the expansion control 1107, thescreen 1100 provides a context snippet 1108, which presents textualcontext taken from the article (or other content) in which the displayedrelationship originally appeared. In some embodiments, the entityrecommendation process of FIG. 8 makes queries of this sort at block804.

FIG. 11B shows search results 1112 returned in response to a secondsearch for all entities related to a specified entity. The example ofFIG. 11B is similar to that of FIG. 11A, except that in FIG. 11B, theuser has entered a query 1111 of the form “Roger Moore” < > * < > *. Inresponse, the content recommendation system has returned search results1112. In some embodiments, the entity recommendation process of FIG. 8makes queries of this sort at block 804.

FIG. 11C shows search results 1122 returned in response to a search forentities in relationships with an entity having a specified facet, wherethe relationships appear in context with one or more specified keyterms.In particular, a user has entered query 1121 (“[actor]< > * < > *context contains “007” or “Bond actor”), which requests all entities inrelationships with an entity that has the facet actor, where therelationships appear in context with either the keyterm “007” or thekeyterm “Bond actor.” In response, the content recommendation system hasreturned the search results 1122. One example relationship record 1123has been expanded to show a context snippet 1124 illustrating thepresence of both keyterms “007” and “Bond actor” in the article thatdescribes the corresponding relationship. Context is typically definedaccording to the query language, for example, as some number ofsurrounding paragraphs, sentences, words, characters, etc. In someembodiments, the entity recommendation process of FIG. 8 makes queriesof this sort at block 808.

FIG. 11D shows search results 1132 returned in response to a firstsearch for entities in relationships with a specified entity, where therelationships appear in context with one or more specified entities. Inparticular, a user has entered query 1131 (“Sean Connery” < > * < > *context contains “Roger Moore”), which requests all entities inrelationships with the entity Sean Connery, where the relationshipsappear in context with the entity “Roger Moore.” In response, thecontent recommendation system has returned the search results 1132including relationship records 1133 and 1135. Relationship record 1133has been expanded to show a context snippet 1134 illustrating thepresence of the entity “Roger Moore” in the article that describes thecorresponding relationship. In some embodiments, the entityrecommendation process of FIG. 8 makes queries of this sort at block810.

FIG. 11E shows search results 1142 returned in response to a secondsearch for entities in relationships with a specified entity, where therelationships appear in context with one or more specified entities. Inparticular, a user has entered query 1141 (“Roger Moore” < > * < > *context contains “Sean Connery”), which requests all entities inrelationships with the entity Roger Moore, where the relationshipsappear in context with the entity “Sean Connery.” In response, thecontent recommendation system has returned the search results 1142including relationship record 1143. Note that relationship record 1143describes the same relationship as relationship record 1135 (FIG. 11D).Typically, during the entity recommendation process, the contentrecommendation system will remove duplicate relationships and/orentities. For example, prior to providing a set of recommended entities,the content recommendation system may remove or coalesce duplicateentities. In some embodiments, the entity recommendation process of FIG.8 makes queries of this sort at block 810.

In the following, additional example techniques for articlerecommendation, entity recommendation, and query construction arediscussed.

Article Recommendation in an Example Embodiment

The following describes an approach to article recommendation used byone example embodiment. Similar techniques could be applied to othermedia, including videos, images, advertisements, and the like.

1. Article Retrieval and Ranking

For the underlying information retrieval capability, the contentrecommendation system of this embodiment uses Lucene, which is anexample of a Vector Space Model (VSM) based information retrievalsystem. Lucene retrieves and ranks documents in an index by determininghow relevant a given document is to a user's query. Additional detailsregarding Lucene's ranking are available at:http://lucene.apache.org/java/2_1_0/scoring.html andhttp://lucene.apache.org/java/2_1_0/api/org/apache/lucene/search/Similarity.html.

In general, the more times a query term appears in a document relativeto the number of times the term appears in all the documents in thecollection, the more relevant that document is to the query. This isaccomplished by using the TF-IDF (“term frequency-inverse termfrequency”) weighting in the scoring of documents.

In addition, Lucene allows influencing of search results by a processknown as “boosting.” The content recommendation system applies documentlevel boosting—while indexing, the function document.setBoost( ) isinvoked before a document is added to the index (such as may berepresented by document store 217 a).

The boost value for each document is computed as a combination of thedocument's source credibility and date.

The content recommendation system prefers to retrieve documents fromcredible sources and documents that are recent. In order to do so, thedocument recommendation system assigns to each document source acredibility score (e.g., 1 to 5), which is translated into a credibilityweight. Document credibility can be assigned manually or computed basedon some properties of a source (e.g., Google PageRank of the source Webpage, Internet traffic to the source Web page, etc.). Credibility weightis multiplied by recency weight to form document boost.

Recency weight is based on the difference between the date of a documentand the date when ingestion started. For example:

TABLE 1 1. int dateDifference = docDate − ingestionStartDate; 2. if (dateDifference < −185 ) dateDifference = −66; 3. // Dateboost for docsolder than 185 days is identical 4. else if ( dateDifference < −35 ) 5.dateDifference = −35 + ( ( dateDifference + 35 ) / 5 ) ; 6. // Dateboostfor docs older than 35 days has 7. // a smaller gradient 8. dateBoost =1.1 {circumflex over ( )} dateDifference 9. documentBoost =credibilityBoost * dateBoost

2. Article Recommendation

The goal of article recommendation is to return a set of articles thatcover the set of entities in a collection. The content recommendationsystem prefers articles that: cover multiple entities than just about asingle entity; provide more recent information or news about theentities; and/or are relevant to the entities (e.g., the articles areabout one or more of the entities, instead of simply mentioning them inpassing).

Also, the content recommendation system prefers that most of theentities are covered by the returned articles, instead of only thedominant entities being covered by the returned articles. To do that,the content recommendation system issues multiple queries. Each timewhen constructing a new query, the content recommendation system removesthe entities that have been well covered by articles from previousqueries. This recursive algorithm is described as below.

Initialization:

Set threshold T=number of requested articles (nReqArticles)/collectionsize.

Set maximum number of recursions=N (currently set to 4).

Start with an empty article set A, and an entity set S that contains allthe entities in the given collection (E1, E2, . . . , Ek).

Step 1:

Given the entity set S={E1, E2, . . . , Ek}, issue a query that is aboolean OR of all the entities: e.g., query(E1) OR query(E2) OR . . . ORquery(Ek). See below for a description of how queries are constructedfor each entity according to one embodiment.

The query will return a list of articles. The order of retrievedarticles is based on information retrieval weighting, described above,taking into account the credibility score, the recency score, termfrequency, number of terms in the article, length of the article, andpresence of terms in the title of the article.

Then place the first nReqArticles articles to set A. The contentrecommendation system does not retrieve all articles returned by thequery because the retrieval of an article is computationally expensive.

If a total number of articles returned for the query is smaller or equalto nReqArticles, jump to Step 3, because shorter queries will not returnadditional articles.

Step 2:

For each entity in set S, count total number of articles in set A thatinclude the entity; and if the count exceeds the threshold T, remove theentity from the set S.

If the entity set S is not empty AND number of recursions is less thanthreshold N, then go to Step 1, otherwise continue to Step 3.

Step 3:

Sort the articles in set A based on a measure that combines recency ofeach article (e.g., publishing date) and number of collection entitiescovered in the article. Return the sorted article list.

As an example of the operation of the above process, if a user wants toretrieve 6 articles for a collection that includes Roger Moore and SeanConnery the content recommendation system submits a query: did123 ANDdid456, where did123 is a unique id for Roger Moore, and did456 is aunique id for Sean Connery. If the underlying information retrievalsystem (e.g., Lucene) returns total of 20 articles the contentrecommendation system retrieves detailed information about the first 6articles. It may be found that 4 articles are about Sean Connery, 1article is about Roger Moore, and 1 article is about both of them. Asthe number of articles for Sean Connery exceeds the threshold T, he isremoved from the set S and a second query is executed: did123. Thecontent recommendation system then combines results of the second querywith the results of the first query, taking care that it does not addarticles present in results for both queries twice.

Entity Recommendation in an Example Embodiment

The following describes an approach to recommending other entities thatmight be of interest to the user, as used by one example embodiment. Thegoal is to suggest other entities that are either related to or similarto the entities in the collection. The content recommendation systemattempts to understand the user's interest based on the existingentities in a collection. Furthermore, the entity suggestion shouldadapt to user actions, such as addition/deletion of entities. Thecontent recommendation system prefers entities that are related tomultiple entities in the collection and/or entities of the same type asthe ones in the collection.

Process Summary

1) Run independent queries for each entity.

2) From search results, determine characteristics of the collection,e.g., main facets, common keyterms, and the like.

3) Run secondary queries to find other entities that are similar.

4) Collect top entities from all query results, put them in a bucket,and count them.

5) Give more weight to entities that match the main facets in thecollection.

Process Details:

Step 1:

Run parallel relationship search queries using each entity in thecollection:

E 1 < >^(*) < >^(*)E 2 < >^(*) < >^(*)… Ek < >^(*) < >^(*)

The relationship query examples are shown in a syntax defined accordingto the relationship query language (EQL) of Evri™ Corporation. Otherqueries as defined by other languages may be similarly incorporated.Again, depending on the collection element type (e.g., a KDE or KNDE),the content recommendation system either uses the entity's ID or name(plus disambiguating terms, if necessary) when constructing the query. Adiscussion of how queries may be constructed for each entity isdescribed further below.

Step 2:

Iterate through the top N relationship search results from each query,and collect the following pieces of information:

1) Facet counts—frequency count of facets assigned to each query entityin the returned relationships. The facet refers to the type or role thatan entity has in a given context. For example, Arnold Schwarzeneggercould be referred to as a politician, an actor, a producer, or a bodybuilder, etc. The facet counts allow the content recommendation systemto identify the main facets shared by the entities in the givencollection.

2) Related entity counts—collect other entities co-occurring in eachreturned relationship, and count their occurrences.

3) Keyterm counts—count frequency of keyterms (e.g., words and phrases)that occur in the returned relationships.

As a result of this step, the content recommendation system has a listof main facets shared by entities in the collection. The facets areranked by their frequency counts.

Then, the keyterms are ranked by a measure of (term-frequency * IDF),where IDF is inverse-document frequency. IDF is a measure of the generalimportance of the term (obtained by dividing the number of all documentsby the number of documents containing the term, and then taking thelogarithm of that quotient). The goal is to identify the keyterms thatare most relevant and specific to the given collection.

Step 3:

Take the most common facets shared by the entities in the collection, aswell as the most common keyterms, in order to find other entities thatare potentially similar. The content recommendation system constructsthe following query:

[Facet1] OR [Facet2] <>* <>* CONTEXT CONTAINS Term1 OR Term2 OR ......OR TermN

The “CONTEXT CONTAINS” parameter indicates the terms must appear withina small window of sentences around the relationship. In one embodiment,the default window size is 3 sentences. Again, the contentrecommendation system iterates through the returned relationships andcollects occurring entities and their frequency counts.

Step 4:

Run parallel joint queries to capture the inter-connection betweenentities in the collection:

E 1 < >^(*) < >^(*)CONTEXT  CONTAINS  E 2  OR  E 3  …  OR  EkE 2 < >^(*) < >^(*)CONTEXT  CONTAINS  E 1  OR  E 3  …  OR  Ek…Ek < >^(*) < >^(*)CONTEXT  CONTAINS  E 1  OR  E 2  …  OR  Ek-1

Again, the content recommendation system collects and counts frequencyof other entities occurring in each returned relationship.

Step 5:

Given all the entities collected from Step 2-4, the contentrecommendation system places all of them in a bucket, and accumulates atotal frequency count for each entity. Then, for each entity, thecontent recommendation system checks whether it shares the same facetwith the ones in the top facet list.

If yes, the entity weight is boosted by: frequency * facet score, wherethe facet score is defined as number of entities in the collection thathave this facet.

Finally, the content recommendation system sorts the entities by theirweights, and returns a ranked list of suggested entities

As one example, the entities in a first collection are Roger Moore andSean Connery. The top entities suggested to the user by the aboveprocess are Pierce Brosnan, Timothy Dalton, and Daniel Craig, all ofwhom have played the character (entity) James Bond.

As another example, the entities in a second collection are Sean Conneryand one of his Bond movies, “Dr. No”. Under the above process, the userwill be provided some of the other Bond movies played by Connery,including “From Russia with Love,” “Gold Finger” and “You Only LiveTwice.” Once the user selects one of the movies to add to thecollection, more similar movies will show up on the top of thesuggestion list, such as “Thunderball.”

Extensions and Variations

As discussed, each user-created collection represents certaininformation interest or topic that users want to keep track. The contentrecommendation system returns the latest news that is relevant to thecollection.

In addition to the described entity-centric profile pages, profile pagesmay be created that are centered around topics, stories, or events.Those topics will be represented as collections.

The definition of collection may be extended to include not only namedentities, but also concepts, key terms and facets. Users will be able toadd concepts (e.g., Global Warming, financial crisis) and/or arbitrarykey terms (words or phrases) to be included in a collection. Theconcepts/key-terms are handled similarly as entities. The abovealgorithms for recommending articles and related entities/concepts stillapply. Allowing key-terms or concepts would let users define informationrequest more precisely. For example, suppose a user is interested infollowing any engagement between China and Russia in terms of oil, acollection could be created including entities China and Russia, as wellas a keyterm “oil.”

In addition, users can add one or more facets to a collection. This isuseful when user is interested in a certain category of entities,instead of specific entities. For example, suppose a user wants to keeptrack of college football Heisman Trophy candidates and winners, acollection could be created as (Heisman Trophy,[College_Football_Player]). When searching for articles, the contentrecommendation system returns articles that contain the term “HeismanTrophy” and mentions of any college football players. When recommendingrelated entities, the content recommendation system may restrict thesuggested entities to belong to the given facets. In this case, thefacets in a collection are treated as filter.

The handling of such extensions for the purpose of query construction isdiscussed further below.

Query Construction in an Example Embodiment

In one example embodiment, a collection may comprise of any combinationof: entities (KDE or KNDE), keyterms, or facets. The following describeson approach to query building for each type of collection members.

1. KDEs (“Known and Disambiguated Entities”)

KDEs are associated with their unique ID in the index. Therefore, duringquery time, KDEs can be searched by their ID. For example, the ID forSean Connery is 136965. The actual query being issued for retrievingdocuments is “did_136965”

2. KNDEs (“Known but Not-disambiguated Entities”)

KNDEs are entities that are included in the Entity Store, but they havenot been disambiguated/identified in the index. During query time, theycan only be searched by their name string.

For entities that have ambiguous name, such as those that share a samename with other entities, context constraints can be included in thequery in order to perform disambiguation. The disambiguating queries arein the form of:

-   -   Name (Entity) AND Disambiguating Terms

In other words, given information known about the KNDE in the entitystore (e.g., properties of the entity), a query can be constructed thatwill return relevant results about the entity.

First, it is determined whether to apply disambiguating terms based onthe uniqueness of an entity's name. If the name is substantially unique,then no disambiguation is needed. The uniqueness is determined based onat least some of the following factors:

1) IDF (“inverse document frequency”) weight of the name, computed froma large corpus; the higher the IDF weight, the more unique the name.

2) If the entity is a person name, determining whether it has a commonfamily name or given name. The common family and given names are readilyavailable from external sources.

3) The total number of entities in the entity store that share the samename

Example pseudo-code for determining whether an entity needsdisambiguation follows:

TABLE 2 1. needDisambiguation(Entity): 2. If isCommonPersonName(Entity)= true, return true; 3. Else if IDF−weight(Entity) > 11, return false;4. Else if IDF−weight(Entity) > 8.5 AND 5.Number_of_entities_with_same_name(Entity) < 3, 6. return false; 7. Elseif is_Emerging_Popular_Entity(entity) = true, 8. return false; 9. Else,return true;

The goal of the function is_Emerging_Popular_Entity(entity) is to catchentities that are popular or suddenly emerging (e.g., Susan Boyle,balloon boy) all over the news. Emerging popular entities typically donot need disambiguation.

Popularity of entities may be determined based on external sources, suchas Wikipedia page view counts (Wikipedia traffic statistics on hourlybasis are publicly available from sources such as http://stats.grok.se),Wikipedia in-bound link counts (i.e., how many pages link to the givenpage), popular queries from major search engines, trending topics fromsources like Twitter, or the like.

The function is_Emerging_Popular_Entity(Entity) would return true if a)its popularity measure exceeds a pre-determined threshold; and b) theentity has highest popularity score compared to other entities that havethe same name. For example, Will Smith the football player might bepopular due to his team is playing the 2010 Super Bowl, but Will Smiththe actor still has higher overall popularity score.

Next, for entities that need disambiguation, disambiguation terms areselected based on the given entity's facet. Suppose the KNDE is WillSmith, the American football player of the NFL's New Orleans Saints. Itis known that football players are usually associated with relatedfacets (e.g., Football teams, Football coaches, Football players) and/orkeywords (e.g., NFL, Super bowl, quarterback, tackle, linebacker,touchdown).

The above related facets and keyterms can be termed “facet profiles.”Facet profiles can be generated automatically off-line, using the KDEsthat have been tagged in the index. For example, the entities that havebeen tagged as football player may be analyzed to determine which facetsand keyterms are mostly correlated to football players. Of course, facetprofiles can also be generated manually by a human, or by a combinationof manual and automated processes.

In the entity store, each KNDE is associated with a number of staticproperties that are extracted from structured or semi-structured sources(e.g., Wikipedia). Some of the properties are links to other entities(e.g., for a football player, the list of teams he has played for, thecollege he went to, birthplace, spouse, etc.). The contentrecommendation system processes the linked entities, and selects theones that match the related facets above. For example, for Will Smith,the content recommendation system would select “New Orleans Saints” thefootball team as a related entity.

As the result, the disambiguation terms include facet-related keywords,plus linked entities. For example, the query for Will Smith could be:

“Will Smith” AND (“New Orleans Saints” OR “NFL” OR “Super bowl” OR“Defensive end” OR “tackle”)

3. Keyterms

If a collection entry is a keyterm (word or phrase), the contentrecommendation system uses the term as keyword query for retrievingdocuments.

4. Facets

When a collection entry is a facet, such as a football player, oneapproach is to use the facet directly in the query. In the Evri QueryLanguage (EQL), the syntax for searching for a facet is using squarebracket, such as [Football_Team]. In the index, tagged/disambiguatedentities are annotated with their facets. Thus, a facet query like[Football_Team] would return articles that contain mentions of anyfootball teams.

This approach may have some drawbacks. The ranking of the retrievedarticles is based on factors such as TF-IDF weighting and articlerecency. Therefore, this approach may result in less relevant articlesdue to:

(a) Articles containing many mentions of different entities with thegiven facet. For example, for the query [Football_Team], articles thatcover many games and teams would get ranked higher, but those articlesare usually less interesting.

b) Articles that are very recent but only mention the facet in passing.

c) For news articles and blogs on the Web, the facet appears in newsheadlines or other boiler-plate part of the web page that are notrelated to the main article.

To address this issue, the content recommendation system may restrictthe facets to the main entities in a given article. In one approach,during indexing of each article, the content recommendation system firstidentifies the top K entities in the article, then collects all thefacets assigned to the top entities, and then annotates the article withthose facets using special tokens (e.g., “facet.football.team”).

Later at search time, given a facet Football_Team, the contentrecommendation system translates it into the query term“facet.football.team.” By doing this, the content recommendation systemobtains articles that tend to be about the facet.

The top entities in an article are ranked and selected by an importancemeasure (e.g., how important an entity is with respect to the main topicof the article), which is computed based on: total number of occurrencesof each entity in the article, giving boost to entities appearing in thearticle title, and giving penalty to entities that are place names orpublisher names.

Example Entity Types

The following Table defines several example entity types in an exampleembodiment. Other embodiments may incorporate different types.

TABLE 3 Person Organization Location Concept Event Product ConditionOrganism Substance

Example Facets

The following Table defines several example facets in an exampleembodiment. Other embodiments may incorporate different facets.

TABLE 4 PERSON actor Evri/Person/Entertainment/Actor PERSON animatorEvri/Person/Entertainment/Animator PERSON cinematographerEvri/Person/Entertainment/Cinematographer PERSON comedianEvri/Person/Entertainment/Comedian PERSON fashion_designerEvri/Person/Entertainment/Fashion_Designer PERSON musicianEvri/Person/Entertainment/Musician PERSON composerEvri/Person/Entertainment/Musician/Composer PERSON producerEvri/Person/Entertainment/Producer PERSON directorEvri/Person/Entertainment/Director PERSON radio_personalityEvri/Person/Entertainment/Radio_Personality PERSONtelevision_personality Evri/Person/Entertainment/Television_PersonalityPERSON author Evri/Person/Entertainment/Author PERSON modelEvri/Person/Entertainment/Model PERSON screenwriterEvri/Person/Entertainment/Screenwriter PERSON playwrightEvri/Person/Entertainment/Playwright PERSON conductorEvri/Person/Entertainment/Conductor PRODUCT filmEvri/Product/Entertainment/Movie PRODUCT television_showEvri/Product/Entertainment/Television_Show PRODUCT albumEvri/Product/Entertainment/Album PRODUCT musicalEvri/Product/Entertainment/Musical PRODUCT bookEvri/Product/Entertainment/Book PRODUCT newspaperEvri/Product/Publication PERSON politicianEvri/Person/Politics/Politician PERSON cabinet_memberEvri/Person/Politics/Cabinet_Member PERSON government_personEvri/Person/Politics/Government_Person PERSON political_party_leaderEvri/Person/Politics/Political_Party_Leader PERSON judgeEvri/Person/Politics/Judge PERSON country_leaderEvri/Person/Politics/Politician/World_Leader PERSONjoint_chiefs_of_staffEvri/Person/Politics/Politician/Joint_Chiefs_of_Staff PERSONwhite_house_staff Evri/Person/Politics/White_House_Staff PERSON activistEvri/Person/Politics/Activist PERSON lobbyistEvri/Person/Politics/Lobbyist PERSON ambassadorEvri/Person/Politics/Ambassador PERSON analyst Evri/Person/AnalystPERSON journalist Evri/Person/Journalist PERSON bloggerEvri/Person/Blogger ORGANIZATION bandEvri/Organization/Entertainment/Band ORGANIZATION political_partyEvri/Organization/Politics/Political_Party ORGANIZATION advocacy_groupEvri/Organization/Politics/Advocacy_Group EVENT film_award_ceremonyEvri/Event/Entertainment/Film_Award_Ceremony EVENT music_award_ceremonyEvri/Event/Entertainment/Music_Award_Ceremony EVENTtelevision_award_ceremonyEvri/Event/Entertainment/Television_Award_Ceremony EVENT court_caseEvri/Event/Politics/Court_Case ORGANIZATION television_networkEvri/Organization/Entertainment/Company/Television_Network ORGANIZATIONmusic_production_companyEvri/Organization/Entertainment/Company/Music_Production_CompanyORGANIZATION film_production_companyEvri/Organization/Entertainment/Company/Film_Production_Company LOCATIONcongressional_district Evri/Location/Politics/Congressional_DistrictLOCATION military_base Evri/Location/Politics/Military_Base ORGANIZATIONcongressional_committeeEvri/Organization/Politics/Congressional_Committee ORGANIZATIONinternational_organizationEvri/Organization/Politics/International_Organization ORGANIZATIONgovernment_agency Evri/Organization/Politics/Government_AgencyORGANIZATION armed_force Evri/Organization/Politics/Armed_ForceORGANIZATION terrorist_organizationEvri/Organization/Politics/Terrorist_Organization ORGANIZATION us_courtEvri/Organization/Politics/US_Court ORGANIZATION cabinet_departmentEvri/Organization/Politics/Cabinet_Department LOCATION continentEvri/Location/Continent LOCATION geographic_regionEvri/Location/Geographic_Region LOCATION country Evri/Location/CountryLOCATION province Evri/Location/Province LOCATION stateEvri/Location/State LOCATION city Evri/Location/City LOCATION us_cityEvri/Location/City LOCATION neighborhood Evri/Location/NeighborhoodLOCATION building Evri/Location/Structure/Building LOCATION islandEvri/Location/Island LOCATION mountain Evri/Location/Mountain LOCATIONbody_of_water Evri/Location/Body_of_Water ORGANIZATIONmedia_companyEvri/Organization/Entertainment/Company/Media_CompanyORGANIZATION haute_couture_houseEvri/Organization/Entertainment/Company/Haute_Couture_House ORGANIZATIONpublishing_companyEvri/Organization/Entertainment/Company/Publishing_Company ORGANIZATIONentertainment_company Evri/Organization/Entertainment/Company CONCEPTfictional_character Evri/Concept/Entertainment/Fictional_CharacterPERSON military_leader Evri/Person/Politics/Military_Leader PERSONmilitary_person Evri/Person/Politics/Military_Person EVENTmilitary_conflict Evri/Event/Politics/Military_Conflict PERSON terroristEvri/Person/Politics/Terrorist PERSON criminal Evri/Person/CriminalPERSON explorer Evri/Person/Explorer PERSON inventorEvri/Person/Technology/Inventor PERSON lawyer Evri/Person/Lawyer PERSONartist Evri/Person/Artist PERSON painter Evri/Person/Artist/PainterPERSON revolutionary Evri/Person/Revolutionary PERSON spiritual_leaderEvri/Person/Spiritual_Leader PERSON philosopher Evri/Person/PhilosopherPERSON anthropologist Evri/Person/Anthropologist PERSON architectEvri/Person/Architect PERSON historian Evri/Person/Historian PERSONeditor Evri/Person/Editor PERSON astronaut Evri/Person/Astronaut PERSONphotographer Evri/Person/Photographer PERSON scientistEvri/Person/Technology/Scientist PERSON economist Evri/Person/EconomistPERSON technology_person Evri/Person/Technology/Technology_Person PERSONbusiness_person Evri/Person/Business/Business_Person PERSON stock_traderEvri/Person/Business/Business_Person/Stock_Trader PERSON first_ladyEvri/Person/Politics/First_Lady ORGANIZATION us_state_legislatureEvri/Organization/Politics/Legislative_Body/State_LegislatureORGANIZATION legislative_bodyEvri/Organization/Politics/Legislative_Body ORGANIZATION executive_bodyEvri/Organization/Politics/Executive_Body PERSON team_ownerEvri/Person/Sports/Team_Owner PERSON sports_announcerEvri/Person/Sports/Sports_Announcer PERSON sports_executiveEvri/Person/Sports/Sports_Executive PERSON olympic_medalistEvri/Person/Sports/Olympic_Medalist PERSON athleteEvri/Person/Sports/Athlete PERSON coach Evri/Person/Sports/Coach PERSONsports_official Evri/Person/Sports/Sports_Official PERSONmotorcycle_driver Evri/Person/Sports/Athlete/Motorcycle_Rider PERSONrace_car_driver Evri/Person/Sports/Athlete/Race_car_Driver ORGANIZATIONauto_racing_team Evri/Organization/Sports/Auto_Racing_Team PERSONbaseball_player Evri/Person/Sports/Athlete/Baseball_Player ORGANIZATIONbaseball_team Evri/Organization/Sports/Baseball_Team PERSONbasketball_player Evri/Person/Sports/Athlete/Basketball_PlayerORGANIZATION basketball_team Evri/Organization/Sports/Basketball_TeamPERSON football_player Evri/Person/Sports/Athlete/Football_PlayerORGANIZATION football_team Evri/Organization/Sports/Football_Team PERSONhockey_player Evri/Person/Sports/Athlete/Hockey_Player ORGANIZATIONhockey_team Evri/Organization/Sports/Hockey_Team PERSON soccer_playerEvri/Person/Sports/Athlete/Soccer_Player ORGANIZATION soccer_teamEvri/Organization/Sports/Soccer_Team ORGANIZATION sports_leagueEvri/Organization/Sports/Sports_League PERSON cricketerEvri/Person/Sports/Athlete/Cricketer ORGANIZATION cricket_teamEvri/Organization/Sports/Cricket_Team PERSON cyclistEvri/Person/Sports/Athlete/Cyclist ORGANIZATION cycling_teamEvri/Organization/Sports/Cycling_Team PERSON volleyball_playerEvri/Person/Sports/Athlete/Volleyball_Player ORGANIZATIONvolleyball_team Evri/Organization/Sports/Volleyball_Team PERSONrugby_player Evri/Person/Sports/Athlete/Rugby_Player ORGANIZATIONrugby_team Evri/Organization/Sports/Rugby_Team PERSON boxerEvri/Person/Sports/Athlete/Boxer PERSON diverEvri/Person/Sports/Athlete/Diver PERSON golferEvri/Person/Sports/Athlete/Golfer PERSON gymnastEvri/Person/Sports/Athlete/Gymnast PERSON figure_skaterEvri/Person/Sports/Athlete/Figure_Skater PERSON horse_racing_jockeyEvri/Person/Sports/Athlete/Horse_Racing_Jockey PERSON lacrosse_playerEvri/Person/Sports/Athlete/Lacrosse_Player ORGANIZATION lacrosse_teamEvri/Organization/Sports/Lacrosse_Team PERSON rowerEvri/Person/Sports/Athlete/Rower PERSON swimmerEvri/Person/Sports/Athlete/Swimmer PERSON tennis_playerEvri/Person/Sports/Athlete/Tennis_Player PERSON track_and_field_athleteEvri/Person/Sports/Athlete/Track_and_Field_Athlete PERSON wrestlerEvri/Person/Sports/Athlete/Wrestler PERSON triathleteEvri/Person/Sports/Athlete/Triathlete EVENT sports_competitionEvri/Event/Sports/Sports_Event/Sporting_Competition EVENT sports_eventEvri/Event/Sports/Sports_Event EVENT olympic_sportEvri/Event/Sports/Olympic_Sports EVENT electionEvri/Event/Politics/Election LOCATION sports_venueEvri/Location/Sports/Sports_Venue ORGANIZATION sports_divisionEvri/Organization/Sports/Sports_Division ORGANIZATIONsports_event_promotion_companyEvri/Organization/Sports/Sports_Event_Promotion_Company ORGANIZATIONsports_organization Evri/Organization/Sports/Sports_OrganizationORGANIZATION company Evri/Organization/Business/Company ORGANIZATIONnews_agency Evri/Organization/Business/Company/News_Agency PRODUCTcell_phone Evri/Product/Technology/Cell_Phone PRODUCT computerEvri/Product/Technology/Computer PRODUCT softwareEvri/Product/Technology/Software PRODUCT video_gameEvri/Product/Technology/Software/Video_Game PRODUCT video_game_consoleEvri/Product/Technology/Video_Game_Console PRODUCT media_playerEvri/Product/Technology/Media_player ORGANIZATION websiteEvri/Organization/Technology/Website ORGANIZATION technology_companyEvri/Organization/Technology/Company PRODUCT magazineEvri/Product/Publication ORGANIZATION financial_services_companyEvri/Organization/Business/Company/Financial_Services_CompanyORGANIZATION radio_networkEvri/Organization/Entertainment/Company/Radio_Network ORGANIZATIONfutures_exchange Evri/Organization/Business/Futures_ExchangeORGANIZATION stock_exchange Evri/Organization/Business/Stock_ExchangeORGANIZATION government_sponsored_enterpriseEvri/Organization/Politics/Government_Sponsored_Enterprise ORGANIZATIONpolitical_organization Evri/Organization/Politics/Political_organizationORGANIZATION labor_union Evri/Organization/Politics/Labor_UnionORGANIZATION nonprofit_corporationEvri/Organization/Business/Company/Nonprofit_Corporation ORGANIZATIONnonprofit_organization Evri/Organization/Nonprofit_OrganizationORGANIZATION national_laboratoryEvri/Organization/Politics/National_Laboratory ORGANIZATIONunified_combatant_commandsEvri/Organization/Politics/Unified_Combatant_Commands ORGANIZATIONresearch_institute Evri/Organization/Research_Institute CONCEPTstock_market_index Evri/Concept/Business/Stock_Market_Index PERSONbusiness_executiveEvri/Person/Business/Business_Person/Business_Executive PERSONcorporate_directorEvri/Person/Business/Business_Person/Corporate_Director PERSON bankerEvri/Person/Business/Business_Person/Banker PERSON publisherEvri/Person/Business/Business_Person/Publisher PERSON us_politicianEvri/Person/Politics/U.S._Politician PERSON nobel_laureateEvri/Person/Nobel_Laureate PERSON chemist Evri/Person/Chemist PERSONphysicist Evri/Person/Physicist ORGANIZATION business_organizationEvri/Organization/Business/Business_Organization ORGANIZATIONconsumer_organization Evri/Organization/Business/Consumer_OrganizationORGANIZATION professional_associationEvri/Organization/Business/Professional_Association PERSON investorEvri/Person/Business/Business_Person/Investor PERSON financierEvri/Person/Business/Business_Person/Financier PERSON money_managerEvri/Person/Business/Business_Person/Money_Manager ORGANIZATIONaerospace_company Evri/Organization/Business/Company/Aerospace_CompanyORGANIZATION advertising_agencyEvri/Organization/Business/Company/Advertising_Company ORGANIZATIONagriculture_companyEvri/Organization/Business/Company/Agriculture_Company ORGANIZATIONairline Evri/Organization/Business/Company/Airline ORGANIZATIONarchitecture_firm Evri/Organization/Business/Company/Architecture_FirmORGANIZATION automotive_companyEvri/Organization/Business/Company/Automotive_Company ORGANIZATIONchemical_company Evri/Organization/Business/Company/Chemical_CompanyORGANIZATION clothing_companyEvri/Organization/Business/Company/Clothing_Company ORGANIZATIONconsulting_company Evri/Organization/Business/Company/Consulting_CompanyORGANIZATION cosmetics_companyEvri/Organization/Business/Company/Cosmetics_Company ORGANIZATIONdefense_company Evri/Organization/Business/Company/Defense_CompanyORGANIZATION distribution_companyEvri/Organization/Business/Company/Distribution_Company ORGANIZATIONgaming_company Evri/Organization/Business/Company/Gaming_CompanyORGANIZATION electronics_companyEvri/Organization/Business/Company/Electronics_Company ORGANIZATIONenergy_company Evri/Organization/Business/Company/Energy_CompanyORGANIZATION hospitality_companyEvri/Organization/Business/Company/Hospitality_Company ORGANIZATIONinsurance_company Evri/Organization/Business/Company/Insurance_CompanyORGANIZATION law_firm Evri/Organization/Business/Company/Law_FirmORGANIZATION manufacturing_companyEvri/Organization/Business/Company/Manufacturing_Company ORGANIZATIONmining_company Evri/Organization/Business/Company/Mining_CompanyORGANIZATION pharmaceutical_companyEvri/Organization/Business/Company/Pharmaceutical_Company ORGANIZATIONrailway_company Evri/Organization/Business/Company/Railway ORGANIZATIONreal_estate_companyEvri/Organization/Business/Company/Real_Estate_Company ORGANIZATIONretailer Evri/Organization/Business/Company/Retailer ORGANIZATIONshipping_company Evri/Organization/Business/Company/Shipping_CompanyORGANIZATION software_companyEvri/Organization/Technology/Company/Software_Company ORGANIZATIONsteel_company Evri/Organization/Business/Company/Steel_CompanyORGANIZATION telecommunications_companyEvri/Organization/Business/Company/Telecommunications_CompanyORGANIZATION utilities_companyEvri/Organization/Business/Company/Utilities_Company ORGANIZATIONwholesaler Evri/Organization/Business/Company/Wholesaler ORGANIZATIONtelevision_production_companyEvri/Organization/Entertainment/Company/Television_Production_CompanyORGANIZATION food_companyEvri/Organization/Business/Company/Food_Company ORGANIZATIONbeverage_companyEvri/Organization/Business/Company/Food_Company/Beverage_CompanyORGANIZATION restaurantEvri/Organization/Business/Company/Food_Company/Restaurant ORGANIZATIONwinery Evri/Organization/Business/Company/Food_Company/Beverage_CompanyEVENT film_festival Evri/Event/Entertainment/Film_Festival ORGANIZATIONfilm_festival Evri/Event/Entertainment/Film_Festival PRODUCT animeEvri/Product/Entertainment/Anime PRODUCT aircraft Evri/Product/AircraftPRODUCT military_aircraft Evri/Product/Aircraft/Military_AircraftPRODUCT vehicle Evri/Product/Vehicle PRODUCT balletEvri/Product/Entertainment/Ballet PRODUCT operaEvri/Product/Entertainment/Opera PRODUCT paintingEvri/Product/Entertainment/Painting PRODUCT songEvri/Product/Entertainment/Single EVENT technology_conferenceEvri/Event/Technology/Technology_Conference CONCEPT legislationEvri/Concept/Politics/Legislation CONCEPT treatyEvri/Concept/Politics/Treaty ORGANIZATION trade_associationEvri/Organization/Business/Trade_Association ORGANIZATIONtechnology_organizationEvri/Organization/Technology/Technology_Organization ORGANIZATIONeducational_institution Evri/Organization/Educational_InstitutionLOCATION museum Evri/Location/Structure/Building/Museum LOCATIONreligious_building Evri/Location/Structure/Building/Religious_BuildingPERSON astronomer Evri/Person/Astronomer PERSON mathematicianEvri/Person/Mathematician PERSON academic Evri/Person/Academic PERSONdancer Evri/Person/Entertainment/Dancer PRODUCT playEvri/Product/Entertainment/Play LOCATION botanical_gardenEvri/Location/Botanical_Garden LOCATION hospitalEvri/Location/Health/Hospital PERSON psychiatristEvri/Person/Health/Psychiatrist PERSON physicianEvri/Person/Health/Physician PERSON nurse Evri/Person/Health/NurseORGANIZATION journalism_organizationEvri/Organization/Journalism_Organization ORGANIZATIONhealthcare_company Evri/Organization/Business/Company/Healthcare_CompanyORGANIZATION religious_organizationEvri/Organization/Religious_Organization PERSON biologistEvri/Person/Scientist/Biologist PERSON biochemistEvri/Person/Scientist/Biochemist PERSON botanistEvri/Person/Scientist/Botanist PERSON poetEvri/Person/Entertainment/Author/Poet PERSON curlerEvri/Person/Sports/Athlete/Curler PERSON biathleteEvri/Person/Sports/Athlete/Biathlete PERSON alpine_skierEvri/Person/Sports/Athlete/Alpine_Skier PERSON cross-country_skierEvri/Person/Sports/Athlete/Cross-country_Skier PERSON freestyle_skierEvri/Person/Sports/Athlete/Freestyle_Skier PERSON lugerEvri/Person/Sports/Athlete/Luger PERSON nordic_combined_skierEvri/Person/Sports/Athlete/Nordic_Combined_Skier PERSON speed_skaterEvri/Person/Sports/Athlete/Speed_Skater PERSON skeleton_racerEvri/Person/Sports/Athlete/Skeleton_Racer PERSON ski_jumperEvri/Person/Sports/Athlete/Ski_Jumper PERSON snowboarderEvri/Person/Sports/Athlete/Snowboarder PERSON bobsledderEvri/Person/Sports/Athlete/Bobsledder PERSON bodybuilderEvri/Person/Sports/Athlete/Bodybuilder PERSON equestrianEvri/Person/Sports/Athlete/Equestrian PERSON fencerEvri/Person/Sports/Athlete/Fencer PERSON hurlerEvri/Person/Sports/Athlete/Hurler PERSON martial_artistEvri/Person/Sports/Athlete/Martial_Artist PERSON canoerEvri/Person/Sports/Athlete/Canoer LOCATION music_venueEvri/Location/Entertainment/Music_Venue LOCATION aquariumEvri/Location/Aquarium LOCATION cemetery Evri/Location/Cemetery LOCATIONnational_park Evri/Location/National_Park LOCATION volcanoEvri/Location/Volcano LOCATION zoo Evri/Location/Zoo LOCATION structureEvri/Location/Structure LOCATION airport Evri/Location/Structure/AirportLOCATION bridge Evri/Location/Structure/Bridge LOCATION hotelEvri/Location/Structure/Hotel LOCATION palaceEvri/Location/Structure/Palace LOCATION monumentEvri/Location/Structure/Monument LOCATION street Evri/Location/StreetLOCATION amusement_park Evri/Location/Amusement_Park LOCATIONunitary_authority Evri/Location/Unitary_Authority PRODUCT drug_brandEvri/Product/Health/Drug_Brand PRODUCT weapon Evri/Product/WeaponPRODUCT missile_system Evri/Product/Weapon/Missile_System PRODUCTfirearm Evri/Product/Weapon/Firearm PRODUCT artilleryEvri/Product/Weapon/Artillery PRODUCT anti-aircraft_weaponEvri/Product/Weapon/Anti-aircraft_Weapon PRODUCT anti-tank_weaponEvri/Product/Weapon/Anti-tank_Weapon PRODUCT biological_weaponEvri/Product/Weapon/Biological_Weapon PRODUCT chemical_weaponEvri/Product/Weapon/Chemical_Weapon CHEMICAL chemical_weaponEvri/Product/Weapon/Chemical_Weapon SUBSTANCE chemical_weaponEvri/Product/Weapon/Chemical_Weapon PRODUCT explosiveEvri/Product/Weapon/Explosive PRODUCT weapons_launcherEvri/Product/Weapon/Weapons_Launcher PERSON chess_playerEvri/Person/Chess_Player PERSON sculptor Evri/Person/Artist/SculptorPRODUCT game Evri/Product/Game ORGANIZATION theater_companyEvri/Organization/Entertainment/Company/Theater_Company PERSONbadminton_player Evri/Person/Sports/Athlete/Badminton_Player PRODUCTnaval_ship Evri/Product/Watercraft/Naval_Ship PRODUCT battleshipEvri/Product/Watercraft/Naval_Ship/Battleship PRODUCT cruiserEvri/Product/Watercraft/Naval_Ship/Cruiser PRODUCT aircraft_carrierEvri/Product/Watercraft/Naval_Ship/Aircraft_Carrier PRODUCT destroyerEvri/Product/Watercraft/Naval_Ship/Destroyer PRODUCT frigateEvri/Product/Watercraft/Naval_Ship/Frigate PRODUCT submarineEvri/Product/Watercraft/Naval_Ship/Submarine PRODUCT cruise_shipEvri/Product/Watercraft/Cruise_Ship PRODUCT yachtEvri/Product/Watercraft/Yacht PRODUCT ocean_linerEvri/Product/Watercraft/Ocean_Liner LOCATION county Evri/Location/CountyPRODUCT symphony Evri/Product/Entertainment/Symphony ORGANIZATIONtelevision_stationEvri/Organization/Entertainment/Company/Television_Station ORGANIZATIONradio_station Evri/Organization/Entertainment/Company/Radio_StationCONCEPT constitutional_amendmentEvri/Concept/Politics/Constitutional_Amendment PERSONaustralian_rules_footballerEvri/Person/Sports/Athlete/Australian_Rules_Footballer ORGANIZATIONaustralian_rules_football_teamEvri/Organization/Sports/Australian_Rules_Football_Team ORGANIZATIONcriminal_organization Evri/Organization/Criminal_Organization PERSONpoker_player Evri/Person/Poker_Player PERSON bowlerEvri/Person/Sports/Athlete/Bowler PERSON yacht_racerEvri/Person/Sports/Athlete/Yacht_Racer PERSON water_polo playerEvri/Person/Sports/Athlete/Water_Polo_Player PERSON field_hockey_playerEvri/Person/Sports/Athlete/Field_Hockey_Player PERSON skateboarderEvri/Person/Sports/Athlete/Skateboarder PERSON polo_playerEvri/Person/Sports/Athlete/Polo_Player PERSON gaelic_footballerEvri/Person/Sports/Athlete/Gaelic_Footballer PRODUCTprogramming_language Evri/Product/Technology/Programming_Language PERSONengineer Evri/Person/Technology/Engineer EVENT cybercrimeEvri/Event/Technology/Cybercrime EVENT criminal_actEvri/Event/Criminal_Act PERSON critic Evri/Person/Critic PERSONpool_player Evri/Person/Pool_Player PERSON snooker_playerEvri/Person/Snooker_Player PERSON competitive_eaterEvri/Person/Competitive_Eater PRODUCT data_storage_mediumEvri/Product/Technology/Data_Storage_Medium PRODUCT data_storage_deviceEvri/Product/Technology/Data_Storage_Device PERSON mountain_climberEvri/Person/Mountain_Climber PERSON aviator Evri/Person/AviatorORGANIZATION cooperative Evri/Organization/Cooperative CONCEPTcopyright_license Evri/Concept/Copyright_License EVENT observanceEvri/Event/Observance PERSON outdoor_sportspersonEvri/Person/Sports/Outdoor_Sportsperson PERSON rodeo_performerEvri/Person/Sports/Rodeo_Performer PERSON sports_shooterEvri/Person/Sports/Athlete/Sports_Shooter CONCEPT awardEvri/Concept/Award CONCEPT entertainment_seriesEvri/Concept/Entertainment/Entertainment_Series PERSON chefEvri/Person/Chef PERSON cartoonist Evri/Person/Entertainment/CartoonistPERSON comics_creator Evri/Person/Entertainment/Comics_Creator PERSONnobility Evri/Person/Nobility PERSON porn_star Evri/Person/Porn_StarPERSON archaeologist Evri/Person/Scientist/Archaeologist PERSONpaleontologist Evri/Person/Scientist/Paleontologist PERSONvictim_of_crime Evri/Person/Victim_of_Crime LOCATION regionEvri/Location/Region PERSON linguist Evri/Person/Linguist PERSONlibrarian Evri/Person/Librarian PERSON bridge_playerEvri/Person/Bridge_Player PERSON choreographerEvri/Person/Entertainment/Choreographer PRODUCT cameraEvri/Product/Technology/Camera PRODUCT publicationEvri/Product/Publication PRODUCT comic Evri/Product/Entertainment/ComicPRODUCT short_story Evri/Product/Entertainment/Short_Story ORGANIZATIONirregular_military_organizationEvri/Organization/Irregular_Military_Organization SUBSTANCEchemical_element Evri/Substance/Chemical_Element SUBSTANCE alkaloidEvri/Substance/Organic_Compound/Alkaloid SUBSTANCE glycosideEvri/Substance/Glycoside SUBSTANCE amino_acid Evri/Substance/Amino_AcidSUBSTANCE protein Evri/Substance/Protein SUBSTANCE enzymeEvri/Substance/Enzyme SUBSTANCE hormone Evri/Substance/Hormone SUBSTANCEhydrocarbon Evri/Substance/Organic_Compound/Hydrocarbon SUBSTANCEinorganic_compound Evri/Substance/Inorganic_Compound SUBSTANCE lipidEvri/Substance/Organic_Compound/Lipid SUBSTANCE steroidEvri/Substance/Organic_Compound/Lipid/Steroid SUBSTANCE moleculeEvri/Substance/Molecule SUBSTANCE polymerEvri/Substance/Molecule/Polymer SUBSTANCE terpene Evri/Substance/OrganicCompound/Terpene SUBSTANCE toxin Evri/Substance/Toxin SUBSTANCEantibiotic Evri/Substance/Health/Antibiotic SUBSTANCE antioxidantEvri/Substance/Health/Antioxidant SUBSTANCE anti-inflammatoryEvri/Substance/Health/Anti-inflammatory SUBSTANCE antiasthmatic_drugEvri/Substance/Health/Antiasthmatic_drug SUBSTANCE anticonvulsantEvri/Substance/Health/Anticonvulsant SUBSTANCE antihistamineEvri/Substance/Health/Antihistamine SUBSTANCE antihypertensiveEvri/Substance/Health/Antihypertensive SUBSTANCE antiviralEvri/Substance/Health/Antiviral SUBSTANCE painkillerEvri/Substance/Health/Painkiller SUBSTANCE PainkillerEvri/Substance/Health/Painkiller SUBSTANCE anestheticEvri/Substance/Health/Anesthetic SUBSTANCE antibodyEvri/Substance/Antibody SUBSTANCE chemotherapeutic_drugEvri/Substance/Health/Chemotherapeutic SUBSTANCE anti-diabetic_drugEvri/Substance/Health/Anti-diabetic SUBSTANCE antianginal_drugEvri/Substance/Health/Antianginal SUBSTANCE muscle_relaxantEvri/Substance/Health/Muscle_relaxant SUBSTANCE hypolipidemic_drugEvri/Substance/Health/Hypolipidemic_Drug SUBSTANCE psychoactive_drugEvri/Substance/Health/Psychoactive_Drug SUBSTANCE vaccineEvri/Substance/Health/Vaccine SUBSTANCE gastrointestinal_drugEvri/Substance/Health/Gastrointestinal_Drug SUBSTANCEerectile_dysfunction_drugEvri/Substance/Health/Erectile_Dysfunction_Drug SUBSTANCEorganometallic_compoundEvri/Substance/Organic_Compound/Organometallic_Compound SUBSTANCE phenolEvri/Substance/Organic_Compound/Phenol SUBSTANCE ketoneEvri/Substance/Organic_Compound/Ketone SUBSTANCE amideEvri/Substance/Organic_Compound/Amide SUBSTANCE esterEvri/Substance/Organic_Compound/Ester SUBSTANCE etherEvri/Substance/Organic_Compound/Ether SUBSTANCE heterocyclic_compoundEvri/Substance/Organic_Compound/Heterocyclic_Compound SUBSTANCEorganic_compound Evri/Substance/Organic_Compound SUBSTANCE carbohydrateEvri/Substance/Organic_Compound/Carbohydrate SUBSTANCE peptideEvri/Substance/Organic_Compound/Peptide SUBSTANCE organohalideEvri/Substance/Organic_Compound/Organohalide SUBSTANCEorganosulfur_compoundEvri/Substance/Organic_Compound/Organosulfur_Compound SUBSTANCEaromatic_compound Evri/Substance/Organic_Compound/Aromatic_CompoundSUBSTANCE carboxylic_acidEvri/Substance/Organic_Compound/Carboxylic_Acid SUBSTANCE nucleic_acidEvri/Substance/Nucleic_Acid SUBSTANCE ion Evri/Substance/Ion ORGANISMcyanobacterium Evri/Organism/Health/Cyanobacterium ORGANISMgram-positive_bacterium Evri/Organism/Health/Gram-positive_BacteriumORGANISM gram-negative_bacteriumEvri/Organism/Health/Gram-negative_Bacterium ORGANISMacid-fast_bacterium Evri/Organism/Health/Acid-fast_Bacterium ORGANISMdna_virus Evri/Organism/Health/DNA_Virus ORGANISM rna_virusEvri/Organism/Health/RNA_Virus CONDITION symptomEvri/Condition/Health/Symptom CONDITION injuryEvri/Condition/Health/Injury CONDITION inflammationEvri/Condition/Health/Inflammation CONDITION diseaseEvri/Condition/Health/Disease CONDITION cancerEvri/Condition/Health/Disease/Cancer ORGANISM medicinal_plantEvri/Organism/Health/Medicinal_Plant ORGANISM poisonous_plantEvri/Organism/Poisonous_Plant ORGANISM herb Evri/Organism/Herb CONCEPTmedical_procedure Evri/Concept/Health/Medical_Procedure ORGANISMbacterium Evri/Organism/Health/Bacterium ORGANISM virusEvri/Organism/Health/Virus ORGANISM horse Evri/Organism/Horse PERSONfugitive Evri/Person/Fugitive ORGANIZATION military_unitEvri/Organization/Politics/Military_Unit ORGANIZATIONlaw_enforcement_agency Evri/Organization/Politics/Law_Enforcement_AgencyLOCATION golf_course Evri/Location/Golf_Course PERSONlaw_enforcement_agent Evri/Person/Politics/Law_Enforcement_Agent PERSONmagician Evri/Person/Entertainment/Magician LOCATIONeducational_institution Evri/Organization/Educational_InstitutionCONCEPT social_program Evri/Concept/Politics/Social_Program EVENTinternational_conference Evri/Event/Politics/International_Conference

All of the above U.S. patents, U.S. patent application publications,U.S. patent applications, foreign patents, foreign patent applicationsand non-patent publications referred to in this specification and/orlisted in the Application Data Sheet, including but not limited to U.S.patent application Ser. No. 13/038,192, entitled “CONTENT RECOMMENDATIONBASED ON COLLECTIONS OF ENTITIES,” filed Mar. 1, 2011; and U.S.Provisional Patent Application No. 61/309,318, entitled “CONTENTRECOMMENDATION BASED ON COLLECTIONS OF ENTITIES,” filed Mar. 1, 2010,are incorporated herein by reference, in their entireties.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of this disclosure. For example, the methods, techniques, andsystems for content recommendation are applicable to otherarchitectures. For example, instead of utilizing a Vector Space Model ofdocument indexing, systems that are programmed to perform naturallanguage processing (e.g., parts of speech tagging) can be employed.Also, the methods, techniques, and systems discussed herein areapplicable to differing query languages, protocols, communication media(optical, wireless, cable, etc.) and devices (such as wireless handsets,mobile communications devices, electronic organizers, personal digitalassistants, portable email machines, game machines, pagers, navigationdevices such as GPS receivers, etc.).

1.-36. (canceled)
 37. A method in a content recommendation computersystem for determining and presenting collections of entities andrelated content, comprising: receiving via a user interface control ofthe content recommendation computer system, user identification of aplurality of named entities; creating or modifying an entity collectionto include the user identified plurality of named entities, the entitycollection presentable on a display of the content recommendationcomputer system and sharable with other users on separate displays ofother computer systems, the entity collection including multiple namedentities each referenced by one or more of multiple documents indexed bythe content recommendation system, the created or modified entitycollection being separate and distinct from the multiple documentsindexed by the content recommendation system; automatically processingthe created or modified entity collection to automatically determine oneor more additional named entities not already included in the entitycollection that are related to the multiple named entities of the entitycollection by determining whether the one or more additional namedentities share semantic characteristics with one or more of the multiplenamed entities or appear in semantic relationships with one or more ofthe multiple named entities in one or more of multiple documents indexedby the content recommendation system; and presenting via a userinterface control the determined one or more additional named entitiesfor possible inclusion in the entity collection.
 38. The method of claim37 wherein the user interface control for receiving the useridentification of the plurality of named entities is separate from theuser interface control for presenting the determined one or moreadditional named entities for possible inclusion in the entitycollection.
 39. The method of claim 37 wherein the automaticallyprocessing the created or modified entity collection to automaticallydetermine one or more additional named entities not already included inthe entity collection that are related to the multiple named entities ofthe entity collection further comprises: automatically processing thecreated or modified entity collection to automatically determine one ormore additional named entities not already included in the entitycollection that are related to the multiple named entities of the entitycollection by determining whether the one or more additional namedentities appear in semantic context with one or more of the multiplenamed entities in one or more of multiple documents indexed by thecontent recommendation system, appearing in context determined asappearing within a specified number of surrounding paragraphs,sentences, words, and/or characters.
 40. The method of claim 37, furthercomprising: automatically processing the created or modified entitycollection to automatically recommend one or more documents indexed bythe content recommendation system that are related to at least one ofthe multiple named entities of the entity collection; and presenting onthe computer system the determined one or more documents as relateddocuments to the entity collection.
 41. The method of claim 40 whereinthe automatically processing the created or modified entity collectionto automatically recommend one or more documents indexed by the contentrecommendation system that are related to at least one of the multiplenamed entities of the entity collection is determined based at least inpart upon one or more of: whether and how often an indexed articlerefers to one or more of the multiple named entities, recency of anindexed article, or a measure of credibility of an indexed article. 42.The method of claim 40 automatically processing the created or modifiedentity collection to automatically recommend one or more documentsindexed by the content recommendation system that are related to atleast one of the multiple named entities of the entity collectionincludes recommending one or more of: an article, a news story, a Webpage, or a blog post.
 43. The method of claim 40 further comprising:ordering the determined one or more recommended documents beforepresenting the one or more recommended documents based at least in partupon document recency and/or document relevance.
 44. The method of claim40, further comprising: ordering the determined one or more recommendeddocuments before presenting the one or more recommended documents basedat least in part on source credibility, wherein source credibility isdetermined by automatically determining a credibility score based uponproperties of a source of the content item.
 45. The method of claim 37wherein the plurality of named entities and the multiple named entitiesare each electronically represented in the content recommendation systemin a data repository.
 46. The method of claim 37 wherein the userinterface control for receiving user identification of the plurality ofnamed entities is a control on a website page
 47. The method of claim 37wherein user indication is provided by specifying a collection name andan indication of the plurality of the named entities.
 48. The method ofclaim 37 wherein the automatically processing the created or modifiedentity collection to automatically recommend one or more documentsindexed by the content recommendation system that are related to atleast one of the multiple named entities of the entity collection bydetermining whether the one or more additional named entities sharesemantic characteristics with one or more of the multiple named entitiesis determined based at least in part upon whether the one or moreadditional named entities share a facet value with at least one of themultiple named entities of the entity collection.
 49. The method ofclaim 37, further comprising: ordering the determined one or moreadditional named entities before presenting the one or more additionalnamed entities via the user interface control based at least in partupon a frequency each additional named entity appears in the indexeddocuments and based at least in part upon whether the determined one ormore additional named entities share a facet value with at least one ofthe multiple named entities of the entity collection.
 50. The method ofclaim 37, further comprising: receiving an indication of at least one ofthe one or more additional named entities for inclusion in the entitycollection; and modifying the entity collection to include the indicatedone or more additional named entities.
 51. The method of claim 37wherein the user interface for receiving user identification of aplurality of named entities is provided via a code module configured toperform the method in a third-party application context.
 52. The methodof claim 37 wherein automatically processing the created or modifiedentity collection to automatically determine one or more additionalnamed entities not already included in the entity collection that arerelated to the multiple named entities of the entity collection includesdetermining one or more additional named entities that each appear in asubject-verb-object relationship described in an document indexed by thecontent recommendation system and that includes one of the multiplenamed entities.
 53. A non-transitory computer-readable medium whosecontents enable a computing system to recommend content, by performing amethod comprising: receiving via a first user interface controlindications of a plurality of named entities: creating or modifying anentity collection, the entity collection including indications ofmultiple named entities including the user identified plurality of namedentities, each of the multiple named entities in the entity collectionreferenced by one or more of multiple documents indexed by the contentrecommendation system, the created or modified entity collection beingseparate and distinct from the multiple documents indexed by the contentrecommendation system; automatically processing the created or modifiedentity collection to automatically determine one or more additionalnamed entities not already included in the entity collection that arerelated to the multiple named entities of the entity collection bydetermining whether the one or more additional named entities sharecharacteristics with one or more of the multiple named entities orappear in semantic relationships with one or more of the multiple namedentities in one or more of multiple documents indexed by the contentrecommendation system; presenting via a second user interface controlthe determined one or more additional named entities for possibleinclusion in the entity collection; automatically processing the createdor modified entity collection to automatically recommend one or moredocuments indexed by the content recommendation system that are relatedto at least one of the multiple named entities of the entity collection;and presenting via a third user interface control the automaticallyrecommended one or more documents indexed by the content recommendationsystem that are related to at least one of the multiple named entitiesof the entity collection.
 54. A computing system configured to recommendcontent, comprising: a memory; and a user interface code module storedon the memory that is configured, when executed, to: receive via a firstuser interface control indications of a plurality of named entities:create or modify an entity collection, the entity collection includingindications of multiple named entities including the user identifiedplurality of named entities, each of the multiple named entities in theentity collection referenced by one or more of multiple documentsindexed by the content recommendation system, the created or modifiedentity collection being separate and distinct from the multipledocuments indexed by the content recommendation system; automaticallyprocess the created or modified entity collection to automaticallydetermine one or more additional named entities not already included inthe entity collection that are related to the multiple named entities ofthe entity collection by determining whether the one or more additionalnamed entities share characteristics with one or more of the multiplenamed entities or appear in semantic relationships with one or more ofthe multiple named entities in one or more of multiple documents indexedby the content recommendation system; present via a second userinterface control the determined one or more additional named entitiesfor possible inclusion in the entity collection; automatically processthe created or modified entity collection to automatically recommend oneor more documents indexed by the content recommendation system that arerelated to at least one of the multiple named entities of the entitycollection; and present via a third user interface control theautomatically recommended one or more documents indexed by the contentrecommendation system that are related to at least one of the multiplenamed entities of the entity collection.
 55. The computing system ofclaim 54 wherein the module is configured to automatically process thecreated or modified entity collection to automatically recommend atleast one of an article, image, advertisement, audio, video, and/or Webpage.
 56. The computing system of claim 54 wherein the module isconfigured to present the determined one or more additional namedentities for possible inclusion in the entity collection and theautomatically recommended one or more documents indexed by the contentrecommendation system to at least one of a desktop computing system, apersonal digital assistant, a smart phone, a mobile application, alaptop computer, and/or a third-party application.